search

fsprojects / FSharp.TypeProviders.SDK

The SDK for creating F# type providers
https://fsprojects.github.io/FSharp.TypeProviders.SDK/
MIT License
298 stars 94 forks source link

Read only metadata section #284

Closed dsyme closed 5 years ago

dsyme commented 5 years ago

This change means that only a ByteFile for the metadata section is kept after cracking DLLs in the ILModuleReader used in type providers.

We should also adjust the PEReader so it doesn't read all bytes, so those are ephemeral

Expected benefit is about 50% reduction in memory usage by TPs built using the TPSDK (since referenced assemblies make up most of the memory usage)

cartermp commented 5 years ago

Expected benefit is about 50% reduction in memory usage by TPs built using the TPSDK (since referenced assemblies make up most of the memory usage)

Let's verify this with a real-world solution (cc @blumu)

dsyme commented 5 years ago

@TIHan I realise now how horrific the previous caching was - comparing full file contents after doing a ReadAllBytes. This PR contains an improvement to at least use GetLastWriteTimeUtc for the comparison and only later call ReadAllBytes if necessary https://github.com/fsprojects/FSharp.TypeProviders.SDK/pull/284/files#diff-7c540ae6a67f14c48d0403e7b021fc86R6646

We sill should move to a non-weak cache though

dsyme commented 5 years ago

@TIHan I've merged this, please integrate and then validate the overall effect of this combined with the stronger cache from #283

blumu commented 5 years ago

@TIHan shared with me a private build of FSharp.Data built with #283. I am testing it with our "real-word" solution and will update this thread with results.