search

fsprojects / Rezoom.SQL

Statically typechecks a common SQL dialect and translates it to various RDBMS backends
MIT License
670 stars 25 forks source link

Increased Resource Usage #16

Open rkosafo opened 7 years ago

rkosafo commented 7 years ago

There appears to be an increase in the resources (memory and processor) used when editing and saving sql files. The behavior has been observed with with VS2017 and VSCode. In the case of VS, there is often the need to restart it when the memory increases and causes the machine to slow down.

Is this a known issue? Are there any workarounds?

rspeele commented 7 years ago

Can you describe the precise steps to repro?

How much is memory increasing on each save?

rkosafo commented 7 years ago

I'll try and monitor the changes. Usually, it goes up to over 1.5GB. Will report back with further details.

rspeele commented 7 years ago

Been thinking about this a bit more and I'm worried that the problem may be fundamental to it being a generative type provider. This involves generating a dynamic assembly and as far as I know, those can't be garbage collected unless they are created in a separate app domain which is then unloaded. I don't see any app domain creation code in the ProvidedTypes code or in the F# compiler so I suspect the generated assemblies never get freed.

If that's true, then every time you change your migration scripts all the types for your queries get regenerated and added to the memory usage of the compiler service. When you have a lot of SQL queries (which is absolutely an intended use of the type provider in my opinion) the incremental jump on each type-regeneration could be significant.

This has me thinking it may be best to make an early breaking change and switch to an erasing type provider. This would make it impossible to directly reference the generated types from C# or VB.NET, so you'd have to write a mapping layer to create real types for use in other languages, but practically speaking that layer will exist in a lot of apps anyway. Of course another big downside is that it would be a pain to rewrite the type provider to be erasing -- it's not a trivial change.

A significant upside of making this change would be that it would make it much easier to target .NET Core. I think in general erasing type providers are better supported since they are by far the majority in the F# ecosystem. It feels like generative type providers -- while I personally prefer them in concept -- are an afterthought.

rkosafo commented 7 years ago

The ability to use this from non F# languages is certainly a plus for having it as a regenerative type. However, if having it as erasing type will help fix the memory issues, then I would say that might be the way to go. I have currently used it for four applications (2 large - tsql, 2 small - sqlite) in production and this is not really an issue for the small ones.

The flexibility that Rezoom provides make it trivial to add additional scripts and this slows the process down. Particularly since the db is being designed with the application. Everything is fine however if there are very little changes to the script afterwards. It is not very ideal but certainly not a show stopper.

That said, if converting to erasing will help make the experience better, I think it will be the best way to go with it. I'm unfortunately not sure of how much a change this will be. Also, introducing breaking changes now might be ideal.

rspeele commented 7 years ago

I think erasing is the way to go forward. It will be a big change though and I can't say when I will get around to it.

Thank you so much for your feedback and interest in this project!

On Sat, Sep 23, 2017 at 9:43 PM, rkosafo notifications@github.com wrote:

The ability to use this from non F# languages is certainly a plus for having it as a regenerative type. However, if having it as erasing type will help fix the memory issues, then I would say that might be the way to go. I have currently used it for four applications (2 large - tsql, 2 small

  • sqlite) in production and this is not really an issue for the small ones.

The flexibility that Rezoom provides make it trivial to add additional scripts and this slows the process down. Particularly since the db is being designed with the application. Everything is fine however if there are very little changes to the script afterwards. It is not very ideal but certainly not a show stopper.

That said, if converting to erasing will help make the experience better, I think it will be the best way to go with it. I'm unfortunately not sure of how much a change this will be. Also, introducing breaking changes now might be ideal.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/rspeele/Rezoom.SQL/issues/16#issuecomment-331680848, or mute the thread https://github.com/notifications/unsubscribe-auth/ABj-WnToeVnWYpnNvVPymNaSr2UDXiqhks5slbPFgaJpZM4O8fdb .