search

AArnott / CodeGeneration.Roslyn

Assists in performing Roslyn-based code generation during a build.
Microsoft Public License
408 stars 59 forks source link

Use McMaster's DotNetCorePlugins #114

Closed amis92 closed 4 years ago

amis92 commented 5 years ago

Summary

Generators will be based on netcoreapp2.0/2.1 and possibly provide a known configuration file, based on https://github.com/natemcmaster/DotNetCorePlugins

Reasoning

The current model of targeting net461/netstandard1.6 is based on supporting both MSBuild and MSBuildCore running some task DLLs, AFAIK. Since some time, the generation is run as an external process via DotNetCliTool mechanism.

A dotnet-codegen exe is invoked, which is a netcoreapp which uses Roslyn to compile the project and loads generator DLLs that then provide the generated source. Which means that no matter what, the generator DLLs will be run as a netcoreapp anyway.

Using the DotNetCorePlugins project removes the burden of loading assemblies and all the dependency hell, and requiring generators to target netcoreapp will remove a lot of framework assemblies otherwise included as part of build output. An additional benefit is using it will unlock more APIs.

A separate benefit is that we can then provide new Sdk versions that reference dotnet-codegen targeting new netcoreapp versions so that generators can use new APIs etc.

No compatibility is lost because netcoreapp will still (via netstandard2.0+ support) support both current targets (netstandard1.6 and net461).

amis92 commented 5 years ago

I've invested quite a bit of research into packaging nugets, dotnet tools, etc etc etc.

In the end, I've arrived at a conclusion: The easiest thing currently to do is to use Global Tool publishing tasks. They provide a build target that takes care of all important things: Project2Project and NuGet dependencies are both embedded as DLLs into published package, it generates deps.json and correctly puts that in the NuGet.

If there are no better ideas, I'll proceed with implementing this approach (it mostly goes down to providing appropriate sample and writing down the README providing guidance on how to package custom generators into NuGets).

Important note: I do plan on keeping backwards compatibility support of older generators as far as possible, including "probing" the old GeneratorAssemblySearchPaths.

AArnott commented 5 years ago

So are you saying each generator would be encouraged to, like dotnet-codegen, become its own global tool?

amis92 commented 5 years ago

Nooo! No, no. 😅 That'd kill every build using more than one.

Plus, to be pedantic, dotnet-codegen is a DotNetCliTool, which is different from dotnet global tools and upcoming local tools.

It'd just be a matter of "reusing" the build pipeline, tasks, targets etc. The end result is, indeed, a NuGet that could probably be installed as the Global Tool, but since it won't have Main method, it won't be runnable.

So, one could probably try to mimic the Dotnet (Global) Tool build targets, but I don't suppose my MSBuild skills would suffice. Plus, the Global Tool build steps are already well defined, somehow documented and supported, so reusing that seems like a good idea.

amis92 commented 5 years ago

The main point is to have custom generators packaged into NuGets as if they were published - e.g. no dependencies, no need to restore etc.

AArnott commented 5 years ago

Ok then. That's a player I've used plenty before. Sounds good.

Pzixel commented 5 years ago

Hey guys.

I still facing this issue.

For instance, I'm getting

AggregateException: One or more errors occurred. (Could not load file or assembly 'Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'. Could not find or load a specific file. (Exception from HRESULT: 0x80131621))
  System.AggregateException: One or more errors occurred. (Could not load file or assembly 'Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'. Could not find or load a specific file. (Exception from HRESULT: 0x80131621)) ---> System.IO.FileLoadException: Could not load file or assembly 'Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'. Could not find or load a specific file. (Exception from HRESULT: 0x80131621)
     at Solidity.Roslyn.SolidityGenerator.GenerateAsync(TransformationContext context, IProgress`1 progress, CancellationToken cancellationToken)
     at CodeGeneration.Roslyn.Engine.DocumentTransform.EnrichingCodeGeneratorProxy.<GenerateRichAsync>d__5.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at CodeGeneration.Roslyn.Engine.DocumentTransform.<TransformAsync>d__1.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at CodeGeneration.Roslyn.Engine.CompilationGenerator.Generate(IProgress`1 progress, CancellationToken cancellationToken)
     --- End of inner exception stack trace ---
     at CodeGeneration.Roslyn.Engine.CompilationGenerator.Generate(IProgress`1 progress, CancellationToken cancellationToken)
     at CodeGeneration.Roslyn.Generate.Program.Main(String[] args)
  ---> (Inner Exception #0) System.IO.FileLoadException: Could not load file or assembly 'Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'. Could not find or load a specific file. (Exception from HRESULT: 0x80131621)
  File name: 'Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'
     at Solidity.Roslyn.SolidityGenerator.GenerateAsync(TransformationContext context, IProgress`1 progress, CancellationToken cancellationToken)
     at CodeGeneration.Roslyn.Engine.DocumentTransform.EnrichingCodeGeneratorProxy.<GenerateRichAsync>d__5.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at CodeGeneration.Roslyn.Engine.DocumentTransform.<TransformAsync>d__1.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at CodeGeneration.Roslyn.Engine.CompilationGenerator.Generate(IProgress`1 progress, CancellationToken cancellationToken)

I just can't use the tool without being able to parse JSON.

I think I have to revert to 4.88 and use transient version of JSON shipped with generator itself because I see no other options.

Maybe there is some workaround until this one gets implemented?

Pzixel commented 5 years ago

I also really need external dependencies since I'm tired working with SyntaxFactory so I'd like to use Microsoft.CodeAnalysis.Editing.SyntaxGenerator which of course produces the same error.

It's a real pain to be limited to write dependencyless code.

AArnott commented 5 years ago

I sympathize, @Pzixel. To set expectations though, I won't have time to work on this project for the next few weeks at least. If you or @amis92 can make this change and send a PR, that would be awesome.

Pzixel commented 5 years ago

I've seen you enhanced pipeline to make errors actual errors instead of warnings. But I still can't upgrade since I need JSON.Net and I only can use one shipped with 0.4.X versions of the package.

Any progress on that? I'd really like to upgrade but now it will break my app.

amis92 commented 5 years ago

Well to be honest, I've been trying to make some progress on that (https://github.com/amis92/CodeGeneration.Roslyn/tree/feature/sdk-plugins) but I've hit some blockers. For example, I don't have a good way of referencing same-solution code generation projects with the proposed approach.

amis92 commented 5 years ago

Thinking out loud, with the upcoming Local Tools, maybe we could migrate the project-tool we use to the new (and supported) scenario. For example, I'd imagine providing a command dotnet codegen add generator <LOCAL-PATH>. It would setup a correct P2P reference.

And then we don't actually use p2p references, but manually build the projects (invoking Msbuild task) and probably actually publish them, and then reference that publish-path.

Pzixel commented 5 years ago

That would be nice.

Do you mind any adhok solution which may not be ideal but work in a short term? For example resolving assembly dependencies manually and invoking nuget manually as well?

amis92 commented 5 years ago

I think for short term you might want to just hack up your own solution. Of course you could then make a PR to show what you came up with, it'll be more than welcome.

If I do something in this direction, I'll rather take more time but do it well.

AArnott commented 5 years ago

We should avoid nested builds. Since the dotnet codegen tool is invoked from a build, it should not invoke a build of its own.

amis92 commented 5 years ago

Hm. Maybe it won't be needed. I'll need to experiment.

Pzixel commented 5 years ago

I hope i'm not very annoying. But I'm still stuck with 0.4.88 which allows me to use JSON.Net in the generator. I hardly need custom dependencies so I hope I will eventually be able to use it.

Another question about if these deps may be marked as compile-time only and not be required for runtime.

cezarypiatek commented 4 years ago

I think I discovered the source of the problem with loading generator dependencies. Please check the #166

Pzixel commented 4 years ago

Did you see recent netcore changes? They could be useful:

Assembly Load Context Improvements Enhancements to AssemblyLoadContext:

Enable naming contexts Added the ability to enumerate ALCs Added the ability to enumerate assemblies within an ALC Made the type concrete – so instantiation is easier (no requirement for custom types for simple scenarios) See dotnet/corefx #34791 for more details. The appwithalc sample demonstrates these new capabilities.

By using AssemblyDependencyResolver along with a custom AssemblyLoadContext, an application can load plugins so that each plugin’s dependencies are loaded from the correct location, and one plugin’s dependencies will not conflict with another. The AppWithPlugin sample includes plugins that have conflicting dependencies and plugins that rely on satellite assemblies or native libraries.