search

mono / t4

T4 text templating engine
Other
394 stars 101 forks source link

Skip writing assembly to disk when using in-process compiler #103

Open mhutch opened 3 years ago

mhutch commented 3 years ago

When using the in-process compiler in non-debug mode we should be able to avoid writing the compiled assembly to disk and Assembly.Load it directly from memory.

(see #101)

jgiannuzzi commented 3 years ago

I would love to help implement this one! How would you go about this? Currently there is a lot of knowledge about files on disk in TemplatingEngine.CompileCode. Should this somehow move into the CodeCompiler implementations instead?

mhutch commented 3 years ago

That's a good point. We don't need a temp directory at all when using Roslyn in process. We can pass the source file in as a string, and get the assembly and pdb back as byte arrays.

One way would be to change CodeCompiler to always take source files as strings and always return byte arrays. The CscCodeCompiler would then be responsible for writing source files to disk before compilation and reading the assembly after compilation.

A downside to this would approach be that we would read assemblies from disk and load them as byte arrays, which is suboptimal, so I think we should allow the CscCodeCompiler to continue to return an assembly file path.

It would also be good to keep the ability to use a temp directory for diagnostic purposes. If we make CodeCompiler responsible for the disk read/write code then this would mean both RoslynCodeCompiler and CscCodeCompiler would have to duplicate it.

Maybe CodeCompiler could have a bool SupportsCompileInMemory property and a CompileInMemory virtual method.

TemplatingEngine.CompileCode would have two code paths - if keeping temp files, or if the compiler doesn't support in-memory compilation, it would do the temp directory setup and source file writes, call the existing CodeCompiler.CompileFile method, and do cleanup. Else, it would call the new CodeCompiler.CompileInMemory method.

BTW - I just realized a couple problems with https://github.com/mono/t4/commit/89a98f81c42c061f973cd2f419c0c179dda48e09. Firstly, it doesn't load the pdb. Secondly, we can't load the assembly into an Assembly object at this point. We need to defer the load to the AssemblyLoadContext or process in which the code will be executed, or we will leak the assembly. We should probably return the assembly from TemplatingEngine.CompileCode as a (byte assemblyData, byte[] pdbData).

mhutch commented 3 years ago

BTW, I just saw your T4.Build project. Have you seen https://github.com/mono/t4/tree/build-targets/Mono.TextTemplating.Build? It looks like we have significant overlap. Mono.TextTemplating.Build doesn't yet do incremental build or parallel build but I think its targets are more complete, and it has tests. You're welcome to take it over if you'd like :)

jgiannuzzi commented 3 years ago

Thanks for the guidance, @mhutch! I'll give a go at implementing it that way and fix the pdb loading and deferred assembly loading at the same time (I think it'll be easier than to retrofit those fixes with the current CodeCompiler interface).

About T4.Build: I experimented a bit with your Mono.TextTemplating.Build after I started my T4.Build project, but had a strong opinion about what I needed ASAP: .NET Core support, incremental build, fast parallel build, multi-targeting support, and cleanup support. This was actually to be used in the ILGPU project which has 41 templates to be processed at build time and which required slow manual commands for anyone not using Visual Studio on Windows. I was thus not concerned about pre-processing templates, session parameters, .NET Framework support, etc.

It also seemed easier to go for a command line tool to not have the constraints of MSBuild. This has the nice side effect of also making it work on .NET Framework as long as you have .NET Core installed somewhere. It ended up being easier to develop those features in my own tool, and allowed me to experiment further. I don't mean to compete with Mono.TextTemplating.Build though! If I find the time, I'd be happy to take another look at it and try to add the T4.Build features to it.

BTW I just released T4.Build 0.2.0 that uses the AOT compiled version of the Roslyn compiler (by loading the DLLs used by csc.dll). This shaves off about 3s of the startup time on my machine when compared to using the NuGet version of the Roslyn assemblies. It's a bit ad-hoc in T4.Build but I would love to bring that back to Mono.TextTemplating. Let me know if you're interested and we can open another issue to discuss further how to implement it properly.