search

louthy / language-ext

C# functional language extensions - a base class library for functional programming
MIT License
6.48k stars 417 forks source link

A very heavy dll of LanguageExt.Core #673

Closed pmrogala closed 4 years ago

pmrogala commented 4 years ago

Hi,

I have a question regarding the LanguageExt.Core: Do you consider to split some functionalities to separate libraries? or make a lighter version of the lib?

I ask because the generated dll weighs 7.5 MB. I think it's a lot, especially when you want to write some small application/library with only, let's say, option and either classes without most of the extensions.

For example there could be a nuget for every type:

and nuget for everything

Quick solution from top of my head.

PS It is my first issue posted here, so taking the opportunity, I would like to thank you for this amazing library. Great work! 😄

louthy commented 4 years ago

I started the work to break the transformer functions from the Core library a few months back: https://github.com/louthy/language-ext/tree/breakout-transformers

It will halve the size of Core. But, I won't be committing that until I have a LanguageExt.CodeGen story for generating transformers 'live', so it's parked for the moment.

I've never quite understood why it's such an issue to have a large DLL size. It doesn't have any runtime performance issues other than memory usage. Are you working on a device that has a tiny amount of RAM and can't afford to lose a few megs?

Your point about having a separate package for each type is a real non-starter. The reason for the scope of features in this library, is to create a joined up ecosystem. Before language-ext I wrote a monads library called csharp-monad. It is almost exactly what you're suggesting. But the problem is that nothing else uses it, and so it's like Unit in System.Reactive, it doesn't get used as a general Unit type because it's not integrated into the .NET BCL.

Map should return Option when you call map.Find. ToSeq() on Option should return a Seq, etc. The joined-up experience of the types is what gives this library its strength, it's a BCL for functional programming, and so it will always be several megs in size.

The cleanest cut is the transformer functions, because there's a clear one-way dependency and the generated code is pretty large.

pmrogala commented 4 years ago

I just don't like the arguments "buy more RAM", "you have 16GB of RAM" etc. If everyone thought like that it would be bad... (for example having 20 libraries, each 8 MB, it is 160 MB already for just the code)

But anyway: thanks for your long response, clarifying what's behind and hope the breakout-transformers will appear in some near future.

louthy commented 4 years ago

I just don't like the arguments

Sorry you don't like that argument, but it is valid. Back in 1985 when I wrote code for computers with a total of 32k of RAM I cared about every byte. But the benefit of having more freedom now is that we can build tools that make the job of writing large/complex applications easier. If I spent most of my time worrying about byte count then I'd spend less time creating actual value. It would be impossible to create anything of modern value if the old mindset prevailed. Should we also write all code in assembly language because it's not wasteful like C# and the CLR?

jltrem commented 4 years ago

There is certainly a need to draw the line somewhere when considering assembly size, but we aren't near it (wherever it might be). I'm using LanguageExt in an Android system that runs several processes concurrently, all using this library. There are plenty of things I need to consider for memory and performance, but LangExt hasn't been a problem.

There are worlds where using LangExt, or even FP in general, is not appropriate. But C# is not likely to be used in those worlds either.

AlexGS74 commented 4 years ago

It's worth mentioning that Microsoft is actively working on features that will allow to repack/treeshake/link an app so it doesn't include any/most of unused code. It's still WIP as far as I understand, right now you can use IL-Linker to strip some of unused code (here). It will get better with time as MS is strongly interested in getting with C# to the IoT stack. Other things you can try is bundling and pre-jit.

louthy commented 4 years ago

Good info @AlexGS74 ! That’s really useful to know :)

WarrenFerrell commented 3 years ago

@louthy This should really be reopened and maybe expanded. LanguageExt does not benefit at all from .NET 5 Member trimming (I don't know enough about it to know why but System libraries seem to be the only ones getting any benefit). When publishing a ReadyToRun app, LanguageExt.Core.dll is 94 MBs making it 55.8% of our entire published package. How much work still needs to get done to make it happen?

WarrenFerrell commented 3 years ago

Update. need to mark the assembly as trimmable. Doing so reduced the ReadyToRun + TrimMode=Link LanguageExt.Core.dll down to 590 KB (we don't use a lot of the LanguageExt library) As I understand, if <IsTrimmable>true</IsTrimmable> is set on the metadata for the LanguageExt.Core project then people won't need to do this.

explicitly marking library from your own project

  <Target Name="ConfigureTrimming" BeforeTargets="PrepareForILLink">
    <ItemGroup>
      <ManagedAssemblyToLink Condition="'%(Filename)' == 'LanguageExt.Core'">
        <IsTrimmable>true</IsTrimmable>
      </ManagedAssemblyToLink>
    </ItemGroup>
  </Target>

Now the question is whether it is safe to trim the LanguageExt library. I get 11 warnings for LanguageExt when publishing trimmed with <SuppressTrimAnalysisWarnings>false</SuppressTrimAnalysisWarnings> LanguageExtWarnings.txt

IDisposable commented 3 years ago

Some of those warnings seem to be related to this behavior (summarized in the mono linker issues) https://github.com/mono/linker/issues/1815

IDisposable commented 3 years ago

The others seem to be just missing suppressions link used https://github.com/dotnet/runtime/issues/42926

vitek-karas commented 3 years ago

I ran into this link from mono/linker#1815. Looking at the list of generated warnings - most of those are valid concern. Please note that mono/linker#1815 is just to validate that the existing behavior of the trimmer is correct/expected (which it is).

In general, please be very careful adding suppressions for these if you want to use the library with trimming enabled. We ourselves ran into several cases where we were sure the suppression is correct, only to realize later on that there's a special case where it's not.

TysonMN commented 3 years ago

Parts of language-ext are highly optimized by emitting MSIL. I don't expect that the trimmer will know what the dependencies are of that MSIL.

vitek-karas commented 3 years ago

@TysonMN The trimmer sees everything as IL only, so if it's in the assembly itself it should not be a problem. If this is where the assembly generates IL at runtime, that can also be OK, it depends how the generation is done.

TysonMN commented 3 years ago

Yes, I mean the MSIL is emitted at runtime

vitek-karas commented 3 years ago

If it would be problematic, the trimmer should produce a warning about it (when enabled)... so if the above list is representative, it's probably OK.

WarrenFerrell commented 2 years ago

Small update on how problematic the growing LanguageExt.Core dll is (7.4 MB with 3., and 10MB with 4.). Brought 3.14.5 LanguageExt into a base common library in a legacy .NET framework application that has 63 projects, causing LanguageExt.Core.dll's to take up 466.2 MB in every local clone of the repository. .NET framework projects obviously cannot benefit from member level trimming.

It would still be quite helpful to separate out the components in LanguageExt.Core to primitives, their extensions, + Prelude BCL extensions Everything else

Jack-Edwards commented 2 years ago

The LanguageExt.Core.dll is still quite large in .NET 6.0. At 11+ MB, it would be the largest file required by my WASM application. After compression the resulting .gz file is still 3+ MB, if I remember correctly. Which is still larger than the compressed dotnet.wasm dependency.

I filed this bug under the dotnet runtime repo thinking there was an error in the AOT compilation process. Unfortunately, it appears there is just too much going on in the code for this to be suitable for AOT-compiled applications. The resulting binary would be massive if I ever allowed my computer to run long enough to finish compiling.

https://github.com/dotnet/runtime/issues/69166

At this point I am only interested in using some of the monad types. It would be convenient to have smaller, more specific packages like Warren suggested.