"Could not load file or assembly" exception randomly when executing code on docker linux container from bamboo CI

[leszek95b]

Exceptions "Could not load file or assembly 'SomeAssemblyName, Version=VersionNumber, Culture=neutral, PublicKeyToken=SomeToken'. The system cannot find the file specified." are thrown randomly (nondeterministic) when executing code using "dotnet test" for some of the assembles.

Project is based on .NET Core 3.1, currently 3.1.201, but issue is reproducible on 3.1.300 as well. Project contains nunit-based automated tests. Issue occurs when executing code on docker linux containers in continuous environment based on Bamboo. Assemblies that appear in exceptions are the ones loaded from NuGet packages. Sample exceptions:

• Could not load file or assembly 'KellermanSoftware.Compare-NET-Objects, Version=4.65.0.0, Culture=neutral, PublicKeyToken=d970ace04cc85217'. The system cannot find the file specified.
• Could not load file or assembly 'Renci.SshNet, Version=2016.1.0.0, Culture=neutral, PublicKeyToken=1cee9f8bde3db106'. The system cannot find the file specified.

With this same container, this same code, this same environment you can make a number of runs and some of them will work well, some of them with throw exception regarding assembly A, some of them regarding assembly B and so on. There is no pattern.

CI task that builds and runs tests is: dotnet test /path/to/project.csproj --logger:"console;verbosity=detailed" --logger:"trx;LogFileName=TestResults.trx" --results-directory:"results/" --settings:"/path/to/settings/file" -- NUnit.Where="cat == SomeNunitCategory"

Docker container is based on Centos 7.5.

What was checked:

1. Every time when there is an exception thrown, "missing" file that dotnet is unable to find exists in output directory (bin/Debug in my case).
2. File has proper permissions, checked with ls -al in CI plan.
3. File is not corrupted, checked with md5sum in CI plan and sum is this same every time, hence binary this file is OK.
4. dotnet test command was executed with variables COREHOST_TRACE=1 COREHOST_TRACE_VERBOSITY=4 and in log files there is just an exception, no other errors related to assemblies dotnet cannot find.
5. Tried loading from code with Assembly.LoadFrom() and in case of failure it throws System.IO.FileNotFoundException with a path to file in message ... and this file indeed exists in this path.

Described problem occurs only on docker linux Bamboo agents, it's not reproducible in Windows, Linux (real/virtual), docker Linux running locally (same container image).

[leszek95b]

Issues occurs from time to time also regarding assemblies included in runtime, not only external packages installed via NuGet.

System.IO.FileNotFoundException : Could not load file or assembly 'System.ServiceModel.Primitives, Version=4.7.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.

Is it possible to enable any debug logs (fusion not available on Linux) to troubleshoot issue?

[simplicbe]

We are getting this issue also very ofter at the moment. We deploy from Azure DevOps. @leszek95b could you find a solution?

In our case

ystem.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types.
Could not load file or assembly 'Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'. The located assembly's manifest definition does not match the assembly reference. (0x80131040)
   at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
   at System.Reflection.RuntimeAssembly.get_DefinedTypes()
   at Microsoft.AspNetCore.Mvc.ApplicationParts.AssemblyPart.get_Types()
   at Microsoft.AspNetCore.Mvc.Controllers.ControllerFeatureProvider.PopulateFeature(IEnumerable`1 parts, ControllerFeature feature)
   at Microsoft.AspNetCore.Mvc.ApplicationParts.ApplicationPartManager.PopulateFeature[TFeature](TFeature feature)
   at Microsoft.AspNetCore.Mvc.ApplicationModels.ControllerActionDescriptorProvider.GetControllerTypes()
   at Microsoft.AspNetCore.Mvc.ApplicationModels.ControllerActionDescriptorProvider.GetDescriptors()
   at Microsoft.AspNetCore.Mvc.ApplicationModels.ControllerActionDescriptorProvider.OnProvidersExecuting(ActionDescriptorProviderContext context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.DefaultActionDescriptorCollectionProvider.UpdateCollection()

[leszek95b]

Hi @simplicbe, I'm still stuck with this issue. Most probably it's caused by a fact that my docker containers running dotnet code are on virtual machines based on VMWare.

Looking at your exception i can see that your problem is different than mine. In your case its related to manifest (I suspect issues with dotnet publish command), in my case it's randomly missing file in filesystem.

[simplicbe]

@leszek95b the curious problem here is, that different files are missing when compiling and deploying the same code. That's why I thought this might be related.

[influento]

I had same issue.. Downgrading whole solution from asp net core 3.1 to 3.0 fixed it. Nothing else helped me to build stable docker container.

[simplicbe]

@Crispried please let me know if you find some other solution, randomly this is happening in my deployments as well. Create a new container from the same codebase helps sometime.

[RTodorov]

Since this issue is still open and I just came across it, the solution in my case was to add Newtonsoft.Json as a direct dependency to my project. The error I was getting was the exact same error as @simplicbe. I didn't have Newtonsoft.Json as a direct dependency, I simply added the latest version and the error was gone.

[simplicbe]

@RTodorov in my cast this was not helping either. But thanks for you comment.

[incumbent]

@simplicbe have you found any solution to solve the problem? Please let me know.

[simplicbe]

@incumbent make sure, that you only have one version of a depenency. Depending on the compiled result, different versions are copied to the output directory. So it appears to work sometimes and sometimes not.

Keep in mind, that under .net core/.net 5/6 dependencies will be added automatically.

[incumbent]

@simplicbe Thanks bro, I checked all the .dll dependencies from the file XXX.deps.json in the dotnet publish output directory, , the result is that all the .dll related dependencies use the same version. Is there any other way to check whether the dependency has the same version?

[simplicbe]

@incumbent so you also checked the dependencies of your direct dependencies? Because this is nested here.

[incumbent]

@simplicbe, I wrote simple code to only analyze the server.deps.json file, and found all the top level dependencies and their sub level dependencies are all the same. I don't know whether if it is enough to judge they all have the same version.

Below is part of the server.deps.json： Server.deps.txt

The frequency of "Could not load file or assembly" is below 1/1000, it's so confused to me, even if the dependency is not correct, the frequency should not be that low.

[simplicbe]

@incumbent ok, that is way lower than my issue was. At my place it was about 20% of the time. Let me think about my solution back in the days and I will write you.