External dependencies

[Pzixel]

Today I encountered a problem when I need some external dependencies (e.g. Newtonsoft.Json). Is there any way to provide those?

When I'm trying to build a generator with dependency I get:

  Solidity.Roslyn -> /Solidity.Roslyn/Solidity.Roslyn/bin/Debug/netstandard2.0/Solidity.Roslyn.dll
  AggregateException: One or more errors occurred. (Could not load file or assembly 'Newtonsoft.Json, Version=11.0.0.0, C
ulture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'. The system cannot find the file specified.
  )
  System.AggregateException: One or more errors occurred. (Could not load file or assembly 'Newtonsoft.Json, Version=11.0
.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'. The system cannot find the file specified.
  ) ---> System.IO.FileNotFoundException: Could not load file or assembly 'Newtonsoft.Json, Version=11.0.0.0, Culture=neu
tral, PublicKeyToken=30ad4fe6b2a6aeed'. The system cannot find the file specified.

     at Solidity.Roslyn.SolidityGenerator.GenerateAsync(TransformationContext context, IProgress`1 progress, Cancellation
Token cancellationToken)
     at CodeGeneration.Roslyn.DocumentTransform.EnrichingCodeGeneratorProxy.<GenerateRichAsync>d__5.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at CodeGeneration.Roslyn.DocumentTransform.<TransformAsync>d__1.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at CodeGeneration.Roslyn.CompilationGenerator.Generate(IProgress`1 progress, CancellationToken cancellationToken)
     --- End of inner exception stack trace ---
     at CodeGeneration.Roslyn.CompilationGenerator.Generate(IProgress`1 progress, CancellationToken cancellationToken)
Solidity.cs(1,1): error CGR001: System.IO.FileNotFoundException: Could not load file or assembly 'Newtonsoft.Json, Versio
n=11.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'. The system cannot find the file specified. [/Solidity.Rosl
yn/Solidity.Roslyn.Example/Solidity.Roslyn.Example.csproj]
     at CodeGeneration.Roslyn.Generate.Program.Main(String[] args)
  ---> (Inner Exception #0) System.IO.FileNotFoundException: Could not load file or assembly 'Newtonsoft.Json, Version=11
.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'. The system cannot find the file specified.

  File name: 'Newtonsoft.Json, Version=11.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'
     at Solidity.Roslyn.SolidityGenerator.GenerateAsync(TransformationContext context, IProgress`1 progress, Cancellation
Token cancellationToken)
     at CodeGeneration.Roslyn.DocumentTransform.EnrichingCodeGeneratorProxy.<GenerateRichAsync>d__5.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at CodeGeneration.Roslyn.DocumentTransform.<TransformAsync>d__1.MoveNext()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at CodeGeneration.Roslyn.CompilationGenerator.Generate(IProgress`1 progress, CancellationToken cancellationToken)

  <---

Build FAILED.

Solidity.cs(1,1): error CGR001: System.IO.FileNotFoundException: Could not load file or assembly 'Newtonsoft.Json, Versio
n=11.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'. The system cannot find the file specified. [/Solidity.Rosl
yn/Solidity.Roslyn.Example/Solidity.Roslyn.Example.csproj]
    0 Warning(s)
    1 Error(s)

Time Elapsed 00:00:06.23

[Pzixel]

So, any thoughts on that?

[LokiMidgard]

Looks similar to #61

The code generator should search for a .deps.json file. This is normally not included in the nuget package. But will be generated on build. Try to add this file to the nuget of your generator.

At least if this is the same error

[Pzixel]

I didn't hear anything similar before. Could you elaborate, please? What steps should I perform.

The most weird thing is that I'm able to compile it locally, but it doesn't work in docker. Have no idea why though.

[Pzixel]

This is the project I'm trying to build: https://github.com/Pzixel/Solidity.Roslyn/tree/feature/nethereum in https://circleci.com/gh/Pzixel/Solidity.Roslyn/38

The most interesting thing is that master does compile (and it uses Json.Net as well).

[LokiMidgard]

I don't have any experience with docker. Where does your generator dll resides?

For me it was the problem, that I had an generator dll that referenced another library through nuget. It worked localy, but when I packed it in a nuget, it would no longer work.

The problem was that my local build had a .deps.json file in the build folder. The generator can use it to find out what DLLs are needed by my library and prepare assembly resolution. But the nuget package was missing so it was not able to correctly resolve the dependencys.

I added following lines to my project, so the .deps.json file is added to nuget: https://github.com/LokiMidgard/NDProperty/blob/60eab62a3072c7a41d71f07e4b78d299af920ef8/NDProperty.Core/NDProperty.Core.csproj#L17-L21

After that my Problem was gone. I would assume that wherever your generator dll is lying around it is missing this deps.json file. you can test it if you can manfully deploy it from your build folder to the deployed location.

[Pzixel]

All dlls resides inside container where it is built. Docker experience is not actually required because it's as simple as install clean OS and try to clone and build stuff. The main idea here is that you don't have any mystic environment issues like "well I dunno, it works for me". But it's not that easy to understand why it doesn't build on clean OS.

---

Unfortunately, your advice didn't work for me. Looking forward.

---

Hmm, I found the evil doer, this commit makes build to fail:

https://github.com/Pzixel/Solidity.Roslyn/commit/495e2fe52a12ebadd1fbab837ac333e987c75221

It seems that this package has some internal dependency on Json.Net. Surprisingly, I actually use Json.Net for myself and it works (you can see DeserializeObject on top of that) that pass.

Any ideas on how it could be fixed? I tried this deps.json stuff but it looks more like BindingRedirect requirement.

[LokiMidgard]

Unfortunately that is the only problem I had in the past with dependency resolution. I was able to found the culprit then debugging the dependency resolution code in the generator framework. But that will be hard if you can't reproduce it on your machine.

[Pzixel]

@LokiMidgard well, I managed to install brand new ubuntu physically on my laptop. And I can say that it doesn't work there as well. So it becomes interesting. I have https://github.com/Pzixel/Solidity.Roslyn , branch feature/nethereum, which builds fine on windows and mac, but it doesn't on clean ubuntu.

Any ideas why it could happen? @AArnott please, give some advice if any. I'm quite lost here.

[amis92]

@Pzixel the error is still the same I suppose?

Some time ago I had an idea to use https://github.com/natemcmaster/DotNetCorePlugins/ library to handle dependency loading in CodeGeneration.Roslyn. I think it'd be a good idea to incorporate the approach presented there. I'll probably do a pull request for it at some point.

[Pzixel]

Any progress on that?

I'd probably like to help because it blocks my project that I want to integrate in the CI process. The problem is that CI is running on ubuntu machine so I cannot get project compiled. It really hurts I can't adopt CodeGeneration and have to hand write all types that should be just generated.

---

That's worth mentioning that it probably ok with direct dependencies, but the interesting moment here is that if fails probably because CodeGeneration itself uses Json.Net (through `Microsoft.Extensions.DependencyModel) so it's a clash of versions (11.x vs 10.x). It may affect this behavior in some subtle ways.

[Pzixel]

Btw, it worth adding other platforms CI tests as well since AppVeyor already supports Linux tests suites: https://www.appveyor.com/docs/getting-started-with-appveyor-for-linux/

[AArnott]

.NET Core doesn't require binding redirects. It will simply load whatever DLL it finds in the directory of the .exe. As for the .deps file, I know nothing about that. The only experience I've had trying to make a .NET Core app extensible was an MSBuild Task (in fact, this project was part of that effort) and it was so very painful I ended up throwing out the idea and going with a dotnet console app which MSBuild spawns. That's why this codegeneration.roslyn project doesn't generate code within the msbuild process any more as it used to.

I can certainly help get the build and tests going on Ubuntu. I have had much more success at that sort of thing. But if any of you can help with making code generators with dependencies work, I'd be happy to review PRs. Even if it takes me several days to respond.

[Pzixel]

Being said in #98 , I have a huge need in having it work in ubuntu since it's default docker host for netcore applications and I'm currently building one. I have a very cool project stuck because I cannot build in on CircleCI.

[Pzixel]

@LokiMidgard See some posts in #98 , I have written several scenarios. In more details, here is

My code: https://github.com/Pzixel/Solidity.Roslyn My build docker image: https://hub.docker.com/r/pzixel/solidity-dotnet/~/dockerfile/ My failed build: https://circleci.com/gh/Pzixel/Solidity.Roslyn/58

The interesting point here is that my master branch builds fine, although it uses Json.Net. But if I add yet another project with Json.Net in its dependencies tree then resolution fails. I can't exactly explain why it's happening this way.

---

But that will be hard if you can't reproduce it on your machine.

I can't reproduce it on windows or mac, but it's 100% reproducible on Ubuntu, on which dotnet-sdk image is based. I'd like to have the code working there too, but I haven't enough experience to debug and resolve the issue myself, so I need your help guys.

[LokiMidgard]

It's some time, since I looked in this projects source code. But I would start with LoadAssembly, ResolveAssembly and Generate. If you can't attach an debugger add more Logging to the methods. Worsecase log before every statement, and write the varibales to the console.

Your exception is somewhere in following block: https://github.com/AArnott/CodeGeneration.Roslyn/blob/38605a454be13f09cb13a8ee944bf7c318510e9d/src/CodeGeneration.Roslyn/CompilationGenerator.cs#L128-L155

And I assume you could get more information what and where the lib try's to load dll's from the first two methods I mentioned. Maybe you can log where it loads the dll from in your working master, and where he try's it to load from in your not working branch.

[Pzixel]

Any update on that? I really have to have to external dependencies in order to make it compile, but it's not really convinient because I can do almost nothing useful.

I also don't really get GeneratorAssemblySearchPaths parameter, maybe it's related?..

[Pzixel]

Okay, I found something.

Just adding one dependency already doesn't allow it to compile: https://github.com/Pzixel/Solidity.Roslyn/pull/3/files That leads to an error: https://circleci.com/gh/Pzixel/Solidity.Roslyn/76?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link

The problem is that when I use Json.Net 9.0 it's ok because it's dependency of CodeGeneration.Roslyn package itself (through DependencyModel 2.0.4). So when I upgrade it it couldn't locate the assembly. And it's truly is not available. How could it be done?

---

Maybe some magic property like GeneratorAssemblySearchPaths is missing? (I don't set it because I have no folder with dlls in it).

[amis92]

Closed in favor of #114