search

fsprojects / Paket

A dependency manager for .NET with support for NuGet packages and Git repositories.
https://fsprojects.github.io/Paket/
MIT License
2.02k stars 520 forks source link

Paket restore fails randomly when "msbuilding" solution in parallel #858

Closed dzendras closed 8 years ago

dzendras commented 9 years ago

There seems to be a problem with paket restore when using msbuild/p:BuildInParallel=true switch. It happens on our build machine randomly, but frequently (every day):

C:\SolutionDir.paket\paket.targets(36,5): error MSB3073: The command ""C:\SolutionDir.paket\paket.exe" restore --references-files "C:\SolutionDir\FooProject\paket.references" " exited with code 1. [C:\SolutionDir\FooProject\FooProject.csproj]

When using paket restore with --log-file (BTW why isn't it available on the paket's commands list displayed when you run paket without any arguments?) I could see the cause of the failures:

packages folder is locked by paket.exe (PID = 4220). Waiting...

Paket failed with: The process cannot access the file 'C:\SolutionDir\packages\paket.locked' because it is being used by another process.

The workaround is to run paket restore before running msbuild. Is it possible to make paket support such a scenario out of the box?

forki commented 9 years ago

We tried to support this scenario, but there is no "before solution build" event in MSBuild. So everything hooks into project build events. The only way out was to try to serialize the requests and therefore all paket.exe are waiting for the first to finish.

It seems the timeout is too short. On Jun 3, 2015 11:52 AM, "Jedrzej Grabowski" notifications@github.com wrote:

There seems to be a problem with paket restore when using msbuild/p:BuildInParallel=true switch. It happens on our build machine randomly, but frequently (every day):

C:\SolutionDir.paket\paket.targets(36,5): error MSB3073: The command ""C:\SolutionDir.paket\paket.exe" restore --references-files "C:\SolutionDir\FooProject\paket.references" " exited with code 1. [C:\SolutionDir\FooProject\FooProject.csproj]

When using paket restore with --log-file (BTW isn't it available on the paket's commands list displayed when you run paket without any arguments?) I could see the cause of the failures:

packages folder is locked by paket.exe (PID = 4220). Waiting...

Paket failed with: The process cannot access the file 'C:\SolutionDir\packages\paket.locked' because it is being used by another process.

The workaround is to run paket restore before running msbuild. Is it possible to make paket supports such a scenario out of the box?

— Reply to this email directly or view it on GitHub https://github.com/fsprojects/Paket/issues/858.

Stift commented 9 years ago

I just recommend to do a paket restore before you start msbuild and start msbuild with /p:RestorePackages=false. This works fine and fast.

dzendras commented 9 years ago

@forki: I was pretty sure that the extra complexity was not worth it. Just asked to confirm :) @Stift: The solution seems reasonable.

blumu commented 8 years ago

Just hitting the same issue. Just out of curiosity: is it really necessary for paket to put a read access lock on this file when it just needs to restore packages?

forki commented 8 years ago

@blumu the lock is needed since we change the packages folder. If we don't serialize the requests then you get write conflicts there.

blumu commented 8 years ago

@forki Is it not an overkill to have a global lock? Would not it be more appropriate to only lock any package directory that needs to be updated? That would significantly reduce the chance of lock contention and make this issue go away completely when the package folder is up to date.

forki commented 8 years ago

yes in that case it would be enough to lock per nuget package.

issue go away completely when the package\ folder is up to date

that should already be no issue. since lock time would be really short and no timeout occurs.

blumu commented 8 years ago

Large nuget packages can take some time to download so that would not completely eliminate contentions. It would still reduce them significantly though.

forki commented 8 years ago

TBH I don't really think it's worth it. If you restore before build then everything should be fine.

blumu commented 8 years ago

Maybe. I would argue that restoring packages before building is just a workaround, not an actual fix. Paket.exe should ideally be just as bullet proof as nuget.exe.

forki commented 8 years ago

How is nuget bullet proof here? What are they doing to solve this? On Mar 10, 2016 19:41, "William Blum" notifications@github.com wrote:

Maybe. I would argue that restoring packages before building is just a workaround, not an actual fix. Paket.exe should ideally be just as bullet proof as nuget.exe.

— Reply to this email directly or view it on GitHub https://github.com/fsprojects/Paket/issues/858#issuecomment-194995901.

blumu commented 8 years ago

Ok maybe I did not choose the right terminology :-) I hope I did not offend anyone here. So nuget may not be "bullet proof" but maybe slightly more robust? I'm not sure how they implement it in nuget.exe but I never had to worry about this until I tried paket. I have many large nuget dependencies in my project and nuget manages to restore packages automatically at build time without a glitch. It would be preferable if paket could handle package restore just as well. Also the current workaround is not obvious and not clearly documented and I'm guessing many new paket users hit the same issue.

Auxiliary question: how would you use the workaround when building under Visual Studio? Do I need to configure the additional switch for every single project in my solution?

forki commented 8 years ago

I looked at the code that acquires the lock. It waits for 25min. Is that what you see? Restore failing after 25min? So I guess something else is wrong.

I might have an idea where this race condition comes from.

blumu commented 8 years ago

No, my build normally takes less than 5 minutes overall (with msbuild running on 10 parallel threads).

forki commented 8 years ago

Ok please retry with 2.52.2 - I changed it slightly and also gave a little more verbose output.

blumu commented 8 years ago

Just tried 2.52.2. Paket does not fail and I don't get the msbuild warning MSB3073 anymore, however I still get a bunch of other warnings from paket itself:

error   Could not acquire lock to c:\src\packages\paket.locked.
  The process cannot access the file 'c:\src\packages\paket.locked' because it is being used by    another process.
  Could not acquire lock to c:\src\packages\paket.locked.
  Trials left: 77.

The solution is still suboptimal though, there really should not be a global lock around the package directory, especially when packages are all already up to date.