search

MirrorNetworking / Mirror

#1 Open Source Unity Networking Library
https://mirror-networking.com
MIT License
5.21k stars 769 forks source link

Assembly Sharing Violations Occuring with Mirror & Burst #2683

Closed Ro4m3r closed 3 years ago

Ro4m3r commented 3 years ago

When using Mirror in combination with Burst compilation, the compiler will crash from time to time (in my case roughly 10% of compiles).

The error message in this case is either something along the lines of Microsoft (R) Visual C# Compiler error CSXXX or a sharing violation originating from Library/ScriptAssemblies, with a different assembly file each time.

Note that I have not 100% narrowed it down to the combination of Mirror and Burst since I cannot easily reproduce this behavior (except continually compiling until it eventually fails).

However, this issue is not present without Mirror in the project, and I have turned off burst compile for the time being, and have yet to see the issue appear, so I'm fairly certain both Mirror and Burst are trying to write the same assemblies.

SoftwareGuy commented 3 years ago

As mentioned in discord, one possible way of dealing with this is to check if we can access the Assembly in Read-Write mode. The easiest implementation would be a try-catch. If an exception is thrown, we simply log "Assembly is in use, retrying shortly" and delay the next compile attempt on the Weaver.

Alternatively we could hook into the Assembly Finished Compiling pipeline and then make Mirror do its magic. Both are valid workarounds in my eyes.

Ro4m3r commented 3 years ago

After compiling some more I have managed to reproduce the error with burst compilation disabled (but burst still present in the project).

I'm not sure what exactly is causing the issue, but waiting until the assemblies can be safely written to should hopefully fix the issue regardless.

Edit: For reference, these are the errors I am getting:

Weaver: stop because compile errors on target UnityEngine.Debug:Log (object) Mirror.Weaver.CompilationFinishedHook:OnCompilationFinished (string,UnityEditor.Compilation.CompilerMessage[]) (at Assets/Mirror/Editor/Weaver/CompilationFinishedHook.cs:95) UnityEditor.Scripting.ScriptCompilation.EditorCompilationInterface:TickCompilationPipeline (UnityEditor.Scripting.ScriptCompilation.EditorScriptCompilationOptions,UnityEditor.BuildTargetGroup,UnityEditor.BuildTarget,string[])

followed by

Microsoft (R) Visual C# Compiler version 3.5.0-dev-20359-01 (8da8ba0c) Copyright (C) Microsoft Corporation. All rights reserved.

error CS0009: Metadata file '<projectPath>\Library\ScriptAssemblies\Unity.Mathematics.Extensions.dll' could not be opened -- The process cannot access the file '<projectPath>\Library\ScriptAssemblies\Unity.Mathematics.Extensions.dll' because it is being used by another process.

ndisseldorp commented 3 years ago

It's worth noting that some popular assets (like https://assetstore.unity.com/packages/tools/terrain/vegetation-studio-pro-131835) require Burst to function. So it's not easy for all projects to stop using it. I imagine this will get worse over time as more assets start to take advantage of Burst.

MrGadget1024 commented 3 years ago

To those that have commented, plus the OP, are your projects set to API Compatibility Level .Net Standard 2.0 or .Net 4.x?

LeafMountain commented 3 years ago

I'm having the same problem. We're using .Net 4.x. Anyone have a working work around?

tatelax commented 3 years ago

Same problem here. Unity 2020 LTS latest. Not using burst.

alesmasiar commented 3 years ago

We are having the same issue in our project, however we are starting to get the error more often (maybe 25% of compiles). It's getting pretty annoying, because the only thing that helps is restarting Unity and that takes a while in a larger project.

Some info: Unity 2020.3.10f1 LTS (but it has been happening since 2020.2, I think). We are using Assembly Definitions for our code (roughly 10 assemblies). The sharing violation happens on a different assembly every time (seems to be random) Burst not installed.

MrGadget1024 commented 3 years ago

We are using Assembly Definitions for our code (roughly 10 assemblies).

Do your AsmDef's reference Mirror as a dependency?

alesmasiar commented 3 years ago

We are using Assembly Definitions for our code (roughly 10 assemblies).

Do your AsmDef's reference Mirror as a dependency?

One of them does.

NuclearCookie commented 3 years ago

We are also running into this issue. I confirm it happens on random assemblies. It occurs more on some computers than on others. (1 team mate has it every 10 minutes, I have it every few hours).

tatelax commented 3 years ago

I tried moving my profile files to a brand new project. No luck.

NuclearCookie commented 3 years ago

I noticed that a real callstack hasn't been provided here yet:

Weave behaviours and messages took292 milliseconds
  ProcessSitesModule Assembly-CSharp.dll elapsed time:00:00:00.0143827
- Finished compile Library/ScriptAssemblies/Assembly-CSharp.dll in 1.903921 seconds
- Starting compile Library/ScriptAssemblies/Assembly-CSharp-Editor.dll

-----Compiler Commandline Arguments:
Filename: C:\Program Files\Unity\Hub\Editor\2020.3.9f1\Editor\Data\Tools\RoslynScripts\unity_csc.bat
Arguments: /noconfig @Temp/UnityTempFile-55e5832697d56e3409cd8035b70b4fcf
Responsefile: Temp/UnityTempFile-55e5832697d56e3409cd8035b70b4fcf Contents: 
/target:library
/out:Temp/Assembly-CSharp-Editor.dll
/nowarn:0169
/nowarn:0649
/refout:Temp/Assembly-CSharp-Editor.dll.ref
/deterministic
/debug:portable
/optimize-
/nostdlib+
/preferreduilang:en-US
/langversion:8.0
... (Skipped for brevity)

-----CompilerOutput:-stdout--exitcode: 1--compilationhadfailure: True--outfile: Temp/Assembly-CSharp-Editor.dll
Microsoft (R) Visual C# Compiler version 3.5.0-dev-20359-01 (8da8ba0c)
Copyright (C) Microsoft Corporation. All rights reserved.

error CS0009: Metadata file 'D:\Projects\zombie-quest\UnityProject-ZombieQuest\Library\ScriptAssemblies\UnityEditor.TestRunner.dll' could not be opened -- The process cannot access the file 'D:\Projects\zombie-quest\UnityProject-ZombieQuest\Library\ScriptAssemblies\UnityEditor.TestRunner.dll' because it is being used by another process.
-----EndCompilerOutput---------------

Weaver: stop because compile errors on target
UnityEngine.StackTraceUtility:ExtractStackTrace ()
UnityEngine.DebugLogHandler:LogFormat (UnityEngine.LogType,UnityEngine.Object,string,object[])
UnityEngine.Logger:Log (UnityEngine.LogType,object)
UnityEngine.Debug:Log (object)
Mirror.Weaver.CompilationFinishedHook:OnCompilationFinished (string,UnityEditor.Compilation.CompilerMessage[]) (at Assets/Plugins/Mirror/Editor/Weaver/CompilationFinishedHook.cs:95)
UnityEditor.Compilation.CompilationPipeline/<>c:<SubscribeToEvents>b__23_3 (UnityEditor.Scripting.ScriptCompilation.ScriptAssembly,UnityEditor.Compilation.CompilerMessage[],UnityEditor.Scripting.ScriptCompilation.EditorScriptCompilationOptions)
UnityEditor.Scripting.ScriptCompilation.EditorCompilation:InvokeAssemblyCompilationFinished (UnityEditor.Scripting.ScriptCompilation.ScriptAssembly,System.Collections.Generic.List`1<UnityEditor.Scripting.Compilers.CompilerMessage>,UnityEditor.Scripting.ScriptCompilation.EditorScriptCompilationOptions)
UnityEditor.Scripting.ScriptCompilation.EditorCompilation/<>c__DisplayClass151_0:<CompileScriptAssemblies>b__5 (UnityEditor.Scripting.ScriptCompilation.ScriptAssembly,System.Collections.Generic.List`1<UnityEditor.Scripting.Compilers.CompilerMessage>)
UnityEditor.Scripting.ScriptCompilation.CompilationTask:Poll ()
UnityEditor.Scripting.ScriptCompilation.EditorCompilation:PollCompilation ()
UnityEditor.Scripting.ScriptCompilation.EditorCompilation:TickCompilationPipeline (UnityEditor.Scripting.ScriptCompilation.EditorScriptCompilationOptions,UnityEditor.BuildTargetGroup,UnityEditor.BuildTarget,string[])
UnityEditor.Scripting.ScriptCompilation.EditorCompilationInterface:TickCompilationPipeline (UnityEditor.Scripting.ScriptCompilation.EditorScriptCompilationOptions,UnityEditor.BuildTargetGroup,UnityEditor.BuildTarget,string[])

(Filename: Assets/Plugins/Mirror/Editor/Weaver/CompilationFinishedHook.cs Line: 95)
tatelax commented 3 years ago

I noticed that each time this happens, there's also a debug log for "Weaver: stop because compile errors on target". Is it possible this weaver is running which causes the file to be used by another process....or is the Weaver stopping because this error happened first? Very philosophical...

1ibrium commented 3 years ago

e are having the same issue in our project, however we are starting to get the error more often (maybe 25% of compiles). It's getting pretty annoying, because the only thing that helps is restarting Unity and that takes a while in a larger project.

Some info: Unity 2020.3.10f1 LTS (but it has been happening since 2020.2, I think). We are using Assembly Definitions for our code (roughly 10 assemblies). The sharing violation happens on a different assembly every time (seems to be random)

ME TOO! For large project, this bug is nightmare!!!! I cant believe anyone can win without burst!

etaxi341 commented 3 years ago

Same Problem. It appears every 5 minutes...

alesmasiar commented 3 years ago

So, I've upgraded Mirror to https://github.com/vis2k/Mirror/releases/tag/v42.0.2 and it actually helped quite a bit. It still does happen sometimes, but the frequency is much lower than before (maybe 5% of compiles and usually only those that take a long time ... I also suspect, that compile time might play a role in this, but I'm not sure).

dipyalov commented 3 years ago

From what I've found out - Burst does its thing in a asynchronous way (separate threads or even processes), so from what I know now - it could be hard to synchronize it. I had an issue when Burst throwed similar errors on Unity's startup - and from what I know, Mirror does weave all the assemblies on startup too - so they may conflict.

As a workaround - I moved my Burst jobs to a separate assembly and patched the Weaver to process only the main assembly - and now it seems to work stable, but I will continue watching.

tatelax commented 3 years ago

This is happening to me even with burst compilation disabled

MultiverseEngineering commented 3 years ago

Please fix this issue! In my project everything is bursted. And weave crash my project again and again!

alesmasiar commented 3 years ago

I think the problem with this bug might be, that it's not really deterministic and fully reproducible.

Unless anybody is willing to share their entire project to the Mirror team to work on this, it might be nearly impossible for them to fix it.

And it might not only be project, but also machine-dependent. I suspect the bug happens more often when the CPU takes a longer time to compile the code. Although my current machine has a i9 9900K, we also do have a lot of code and the compile iterations can take 30 - 60 seconds in some cases.

Unfortunately, I am not able to share our project, but if devs need any help trying to pinpoint the issue, I'll gladly assist.

MultiverseEngineering commented 3 years ago

I think the problem with this bug might be, that it's not really deterministic and fully reproducible.

Unless anybody is willing to share their entire project to the Mirror team to work on this, it might be nearly impossible for them to fix it.

And it might not only be project, but also machine-dependent. I suspect the bug happens more often when the CPU takes a longer time to compile the code. Although my current machine has a i9 9900K, we also do have a lot of code and the compile iterations can take 30 - 60 seconds in some cases.

Unfortunately, I am not able to share our project, but if devs need any help trying to pinpoint the issue, I'll gladly assist.

you are right. My project takes longer time to compile. And I can 100% reproduce this bug by add/remove [BurstCompile]. Sometimes crash.

miwarnec commented 3 years ago

hey guys. can anyone add a step by step instruction how to reproduce it please? still working on NetworkTransform V2, didn't have time to look into the burst issue yet.

miwarnec commented 3 years ago

Same problem here. Unity 2020 LTS latest. Not using burst.

you mentioned it happens without burst. could you let us know how to reproduce it?

please guys. any bug that can be reproduced, can be fixed. but you need to let us know how to reproduce it step by step. we can't see your screen from here :)

NuclearCookie commented 3 years ago

Hey @vis2k This one is not easy because it seems to occur at different rates on different computers. We have 1 team member who has this every 5 minutes, while some others only have this each hour or only once per day. The issue seems to sometimes occur when we compile (CompilationFinishedHook.cs) but it's not a 100% reproducable issue. We do have burst in our project but this issue even occurs when burst is disabled. The module that is being accessed both by the compiler and by the weaver (at least I think that's what's going on) is different every time.

We'll try to comment out some code here and there in order to have a better grasp of what is causing this issue.

tatelax commented 3 years ago

Same problem here. Unity 2020 LTS latest. Not using burst.

you mentioned it happens without burst. could you let us know how to reproduce it?

please guys. any bug that can be reproduced, can be fixed. but you need to let us know how to reproduce it step by step. we can't see your screen from here :)

Originally I thought that I was not using Burst but actually, I was.

Since I've updated Mirror to the latest version and disabled Burst, so far it has not happened to me again. I will update this issue if it does happen.

tatelax commented 3 years ago

Just had it happen to me with Burst "Enable Compilation" disabled. I can say it is happening way less frequently after updating to the latest Mirror version.

tatelax commented 3 years ago

I'm still having this happen and I'm willing to invite @vis2k to my repo, join a Discord call, whatever it takes.

tatelax commented 3 years ago

Is there any way to completely disable the Weaver on recompile? I want to disable it and see if this helps.

miwarnec commented 3 years ago

sorry guys, still didn't have time. posting this here for later. https://forum.unity.com/threads/how-does-unity-do-codegen-and-why-cant-i-do-it-myself.853867/

imerr commented 3 years ago

I've recently upgraded to 2020.3 and now I'm seeing the issue (rarely) too Project is using burst

tatelax commented 3 years ago

I completely removed Burst from my project and the issue is still happening.

tatelax commented 3 years ago

I was asked to post a stack trace next time this happened, so here it is. Not much information to go off of. Every time this error happens, it's a different file.

Copyright (C) Microsoft Corporation. All rights reserved.

error CS0009: Metadata file 'E:\hero-rescuers\game\Library\ScriptAssemblies\NWH.Common.NUI.dll' could not be opened -- The process cannot access the file 'E:\hero-rescuers\game\Library\ScriptAssemblies\NWH.Common.NUI.dll' because it is being used by another process.
Assets\Scripts\Gameplay\Vehicles\SyncVolumetricLightWithUnityLight.cs(6,33): warning CS0108: 'SyncVolumetricLightWithUnityLight.light' hides inherited member 'Component.light'. Use the new keyword if hiding was intended.
miwarnec commented 3 years ago

Weaving via ILPostProcessor should fix this: https://github.com/vis2k/Mirror/pull/2891 PR is almost finished. please test if you get a chance and report back.

tatelax commented 3 years ago

I will be testing this fix and will report back if it is resolved.

miwarnec commented 3 years ago

Unity 2020 + latest mirror should fix this. closing unless someone encounters it again:)

anafletcher88 commented 5 months ago

Try using longpath tool, it is effective.

MartinMiller23 commented 5 months ago

Longpath tool is effective for longpath files.