"File already exists" error when installing multiple copies of the same package external to the packages folder, ignores --force

[galaxystar]

This issue is somewhat related to: https://github.com/fsprojects/Paket/issues/1472

When Paket installs to a path outside of the ./packages directory, I receive the error below, when multiple packages reference the same package (and all of which trying to install to the path outside of the ./packages folder)

I've tried using paket install -f and paket install --hard and paket instal -f --hard but they all fail when trying to extract the file.

Regardless of the use case that caused this It would be nice if the force flag would overwrite files which already exist, as I would expect FORCE to do.

Paket failed with:
    Error during extraction of /<path>/packages/<package group>/PackageName/PackageName.1.0.8.nupkg.
Message: Could not create file "/<path outside of packages foder>/<some file>". File already exists.

Stack Trace:

StackTrace:
    at Paket.NuGetV2+ExtractPackage@356-7.Invoke (System.String message) <0x1108c52a0 + 0x0002f> in <filename unknown>:0 
  at Microsoft.FSharp.Core.PrintfImpl+StringPrintfEnv`1[TResult].Finalize () <0x10abe5c10 + 0x00039> in <filename unknown>:0 
  at Microsoft.FSharp.Core.PrintfImpl+Final4@259[TState,TResidue,TResult,A,B,C,D].Invoke (Microsoft.FSharp.Core.FSharpFunc`2 env, A a, B b, C c, D d) <0x1108c50d0 + 0x001bb> in <filename unknown>:0 
  at Microsoft.FSharp.Core.OptimizedClosures+Invoke@3307-3[T2,T3,T4,T5,TResult,T1].Invoke (T2 u, T3 v, T4 w, T5 x) <0x1108c5080 + 0x00047> in <filename unknown>:0 
  at Microsoft.FSharp.Core.OptimizedClosures+Invoke@3301-2[T2,T3,T4,TResult,T1].Invoke (T2 u, T3 v, T4 w) <0x1108c5030 + 0x0003d> in <filename unknown>:0 
  at Microsoft.FSharp.Core.OptimizedClosures+Invoke@3266-1[T2,T3,TResult,T1].Invoke (T2 u, T3 v) <0x10f502eb0 + 0x00033> in <filename unknown>:0 
  at Microsoft.FSharp.Core.OptimizedClosures+Invoke@3253[T2,TResult,T1].Invoke (T2 u) <0x10abe58a0 + 0x00029> in <filename unknown>:0 
  at Paket.NuGetV2+ExtractPackage@356-11.Invoke (System.String arg40) <0x1108c5000 + 0x00021> in <filename unknown>:0 
  at Microsoft.FSharp.Core.FSharpFunc`2[T,TResult].InvokeFast[V] (Microsoft.FSharp.Core.FSharpFunc`2 func, Microsoft.FSharp.Core.T arg1, Microsoft.FSharp.Core.TResult arg2) <0x10f3d6b80 + 0x000ad> in <filename unknown>:0 
  at Microsoft.FSharp.Core.FSharpFunc`2[T,TResult].InvokeFast[V,W] (Microsoft.FSharp.Core.FSharpFunc`2 func, Microsoft.FSharp.Core.T arg1, Microsoft.FSharp.Core.TResult arg2, Microsoft.FSharp.Core.V arg3) <0x1108c4d40 + 0x0016f> in <filename unknown>:0 
  at Microsoft.FSharp.Core.FSharpFunc`2[T,TResult].InvokeFast[V,W,X] (Microsoft.FSharp.Core.FSharpFunc`2 func, Microsoft.FSharp.Core.T arg1, Microsoft.FSharp.Core.TResult arg2, Microsoft.FSharp.Core.V arg3, Microsoft.FSharp.Core.W arg4) <0x1108c48f0 + 0x0023b> in <filename unknown>:0 
  at Paket.NuGetV2+ExtractPackage@349-6.Invoke (System.Exception _arg1) <0x1108c3e10 + 0x00274> in <filename unknown>:0 
  at Microsoft.FSharp.Control.AsyncBuilderImpl+tryWithExnA@893[a].Invoke (System.Runtime.ExceptionServices.ExceptionDispatchInfo edi) <0x1108c3c40 + 0x0003b> in <filename unknown>:0 
  at Microsoft.FSharp.Control.AsyncBuilderImpl+callA@851[b,a].Invoke (Microsoft.FSharp.Control.AsyncParams`1 args) <0x10f5b77b0 + 0x0015f> in <filename unknown>:0 
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () in <filename unknown>:line 0
   at Microsoft.FSharp.Control.AsyncBuilderImpl.commit[a] (Microsoft.FSharp.Control.Result`1 res) in <filename unknown>:line 0
   at Microsoft.FSharp.Control.CancellationTokenOps.RunSynchronously[a] (CancellationToken token, Microsoft.FSharp.Control.FSharpAsync`1 computation, Microsoft.FSharp.Core.FSharpOption`1 timeout) in <filename unknown>:line 0
   at Microsoft.FSharp.Control.FSharpAsync.RunSynchronously[T] (Microsoft.FSharp.Control.FSharpAsync`1 computation, Microsoft.FSharp.Core.FSharpOption`1 timeout, Microsoft.FSharp.Core.FSharpOption`1 cancellationToken) in <filename unknown>:line 0
   at Paket.InstallProcess+CreateModel@151-1.Invoke (KeyValuePair`2 kv') in <filename unknown>:line 0
   at Microsoft.FSharp.Collections.IEnumerator+map@111[b,a].DoMoveNext (b& ) in <filename unknown>:line 0
   at Microsoft.FSharp.Collections.IEnumerator+MapEnumerator`1[T].System-Collections-IEnumerator-MoveNext () in <filename unknown>:line 0
   at Microsoft.FSharp.Core.CompilerServices.RuntimeHelpers.takeOuter@713[T,TResult] (Microsoft.FSharp.Core.CompilerServices.ConcatEnumerator`2 x, Microsoft.FSharp.Core.Unit unitVar0) in <filename unknown>:line 0
   at Microsoft.FSharp.Core.CompilerServices.RuntimeHelpers.takeInner@706[T,TResult] (Microsoft.FSharp.Core.CompilerServices.ConcatEnumerator`2 x, Microsoft.FSharp.Core.Unit unitVar0) in <filename unknown>:line 0
   at Microsoft.FSharp.Core.CompilerServices.RuntimeHelpers.takeOuter@713[T,TResult] (Microsoft.FSharp.Core.CompilerServices.ConcatEnumerator`2 x, Microsoft.FSharp.Core.Unit unitVar0) in <filename unknown>:line 0
   at Microsoft.FSharp.Core.CompilerServices.RuntimeHelpers.takeInner@706[T,TResult] (Microsoft.FSharp.Core.CompilerServices.ConcatEnumerator`2 x, Microsoft.FSharp.Core.Unit unitVar0) in <filename unknown>:line 0
   at Microsoft.FSharp.Core.CompilerServices.RuntimeHelpers.takeOuter@713[T,TResult] (Microsoft.FSharp.Core.CompilerServices.ConcatEnumerator`2 x, Microsoft.FSharp.Core.Unit unitVar0) in <filename unknown>:line 0
   at Microsoft.FSharp.Core.CompilerServices.RuntimeHelpers.takeInner@706[T,TResult] (Microsoft.FSharp.Core.CompilerServices.ConcatEnumerator`2 x, Microsoft.FSharp.Core.Unit unitVar0) in <filename unknown>:line 0
   at Microsoft.FSharp.Core.CompilerServices.RuntimeHelpers+ConcatEnumerator`2[T,TResult].System-Collections-IEnumerator-MoveNext () in <filename unknown>:line 0
   at System.Collections.Generic.List`1[T]..ctor (IEnumerable`1 collection) in <filename unknown>:line 0
   at Microsoft.FSharp.Collections.SeqModule.ToArray[T] (IEnumerable`1 source) in <filename unknown>:line 0
   at Paket.InstallProcess.CreateModel (System.String root, Boolean force, Paket.DependenciesFile dependenciesFile, Paket.LockFile lockFile, Microsoft.FSharp.Collections.FSharpSet`1 packages) in <filename unknown>:line 0
   at Paket.InstallProcess.InstallIntoProjects (Paket.InstallerOptions options, Boolean forceTouch, Paket.DependenciesFile dependenciesFile, Paket.LockFile lockFile, Microsoft.FSharp.Collections.FSharpList`1 projectsAndReferences) in <filename unknown>:line 0
   at Paket.UpdateProcess.SmartInstall (Paket.DependenciesFile dependenciesFile, Paket.UpdateMode updateMode, Paket.UpdaterOptions options) in <filename unknown>:line 0
   at <StartupCode$Paket-Core>.$PublicAPI+Install@164.Invoke (Microsoft.FSharp.Core.Unit unitVar0) in <filename unknown>:line 0
   at Paket.Utils.RunInLockedAccessMode[a] (System.String rootFolder, Microsoft.FSharp.Core.FSharpFunc`2 action) in <filename unknown>:line 0

[galaxystar]

I've narrowed this issue down to ExtractPackage in NuGetV2.fs, on line 351 where Paket uses ZipFile.ExtractToDirectory. We could use ZipArchive instead (which is used elsewhere in the project). If the Force flag is set then use code similar to what is provided here: http://stackoverflow.com/questions/14795197/forcefully-replacing-existing-files-during-extracting-file-using-system-io-compr

[forki]

do you have a repro sample for me?

[galaxystar]

A simple repro should be as follows:

Create a package that contains 1 dependency packet. The dependent package should have at least one file, and map its content to be deployed outside of the package folder. Use paket install to install the package. Then use paket install --force to try and install the package again, observe the following message:

Paket failed with:
    Error during extraction of /<path>/packages/<package group>/PackageName/PackageName.1.0.8.nupkg.
Message: Could not create file "/<path outside of packages foder>/<some file>". File already exists.

Package A depends on Package B

Package A paket.dependency file:

...
    nuget PackageB
...

Package B paket.template file:

...
files
    bin/ ==> ../../<some_path_outside_packages_folder>/
...

[forki]

and map its content to be deployed outside of the package folder

Is this something that nuget supports? That sounds pretty dangerous to me

[galaxystar]

Paket supports it, and I rely on it. It's a bit of a round about way to get what I want. I'm using it to deploy a bootstrapper script into a folder that can then later deploy (copy) the package groups where I want them.

My ideal solution would be to be able to Install packages and then be able to Deploy package groups where I want them, in 2 separate steps. It would be nice to be able to map the group's deploy location in the template file. At which point I could deploy to paths outside of the packages folder.

p.s. I know this is not exactly how Paket was intended to be used, but my toolset (Unity3d) does not support the "proper way" to use Paket. And no, I don't want to use Paket.Unity3D.

[forki]

Paket supports it, and I rely on it.

It's more an accidental feature. I don't think it's "supported"

[galaxystar]

We fixed a bug (https://github.com/fsprojects/Paket/issues/1472) a few weeks back with globbing when packaging which allowed me to continue to use it . Sounds "supported" :-).

But the point of the matter, this repro case may not be "supported", but the request is still sound. When using the "--force" flag, it should override any existing files, which it is not doing.

[forki]

just tried to reproduce. see 65bb4a8

the message says: "Can't extract since it would create a file outside of target folder"

so what am I doing differently?

[galaxystar]

Perhaps Paket 3 handles it differently. My test case is using Paket 2.51.4.0 (sorry for not mentioning it sooner).

[forki]

nope the test is against paket 2

[galaxystar]

The screenshot you posted says "Paket Version 3.0.0.0"

[forki]

Yes, but the test case is showing same error in both branches. On Mar 9, 2016 20:25, "galaxystar" notifications@github.com wrote:

The screenshot you posted says "Paket Version 3.0.0.0"

— Reply to this email directly or view it on GitHub https://github.com/fsprojects/Paket/issues/1504#issuecomment-194464691.

[galaxystar]

I'm not sure, I just installed Paket 2.51.11.0 and I see an identical error to what I posed above.

Paket failed with:
    Error during extraction of /Users/galaxystar/Git/Fyber/packages/bootstrap/PackageManagerBootStrap/PackageManagerBootStrap.1.0.7.nupkg.
Message: Could not create file "/Users/galaxystar/Git/Fyber/Assets/Plugins/Packages/bootstrap/PackageManagerBootStrap/Editor/PackageManager.cs". File already exists.
 In rare cases a firewall might have blocked the download. Please look into the file and see if it contains text with further information.

[forki]

Can you please try to look at my sample? Can you reproduce with that sample? It's basically what you described above, but maybe there is a significant difference. On Mar 9, 2016 20:30, "galaxystar" notifications@github.com wrote:

I'm not sure, I installed Paket 2.51.4.0 and I see an identical error to what I posed above.

— Reply to this email directly or view it on GitHub https://github.com/fsprojects/Paket/issues/1504#issuecomment-194468071.

[galaxystar]

The names paket.A.templatetemplate and paket.A.templatetemplate, which I assume are different for the integration testing system.

Paket.lock file contains packages C and D (I'm not sure where they came from). And the Paket.lock seems to have gotten the dependency order mixed up. It looks like the order of dependencies are a little wrong. the paket.dependencies for Package B, says it's dependent on Package A, and in the Package A Template file it says it has a dependency for Package B. This appears to be cyclic.

Beyond that, the files section in the paket.template looks fine. However with my use case, I did a directory not just a single file. try to replace the files section with this:

files
     /files/ ==> ../../temp/ 

Where the test.txt file is in the /files/ directory.

[forki]

paket.A.templatetemplate will be replaced to paket.A.template during prepare phase in the untegration test. This is just to shield against paket commands from outter levels.

You are right. the lock file ist wrong, but we call paket update so it's not important. I fixed that anyways.

I also changed it to folders.

The following zip conatins the situation after packaging. What do you get if you run paket update in that folder?

repro.zip

[galaxystar]

Hello, Here's my results from the test

$ paket install --force

Paket version 2.51.11.0
Resolving packages for group Main:
 - Paket.Test.A 1.0.0
 - Paket.Test.B 1.0.0
Locked version resolution written to /Users/galaxystar/Downloads/repro/paket.lock
0 seconds - ready.

Now running it a second time... $ paket install --force

Paket version 2.51.11.0
Skipping resolver for group Main since it is already up-to-date
/Users/galaxystar/Downloads/repro/paket.lock is already up-to-date
Paket failed with:
    Error during extraction of /Users/galaxystar/Downloads/repro/packages/Paket.Test.B/Paket.Test.B.1.0.0.nupkg.
Message: Could not create file "/Users/galaxystar/Downloads/repro/outerFolder/test.txt". File already exists.

[forki]

Mhm I need to take a look if mono is doing this differently On Mar 10, 2016 19:53, "galaxystar" notifications@github.com wrote:

Hello, Here's my results from the test

$ paket install --force

Paket version 2.51.11.0 Resolving packages for group Main:

• Paket.Test.A 1.0.0
• Paket.Test.B 1.0.0 Locked version resolution written to /Users/galaxystar/Downloads/repro/paket.lock 0 seconds - ready.

Now running it a second time... $ paket install --force

Paket version 2.51.11.0 Skipping resolver for group Main since it is already up-to-date /Users/galaxystar/Downloads/repro/paket.lock is already up-to-date Paket failed with: Error during extraction of /Users/galaxystar/Downloads/repro/packages/Paket.Test.B/Paket.Test.B.1.0.0.nupkg. Message: Could not create file "/Users/galaxystar/Downloads/repro/outerFolder/test.txt". File already exists.

— Reply to this email directly or view it on GitHub https://github.com/fsprojects/Paket/issues/1504#issuecomment-194999599.

[OnurGumus]

Any updates ? I am just having this issue when I run paket convert-from-nuget.

[matthid]

This sounds like depending on https://github.com/snyk/zip-slip-vulnerability

You really should reconsider your workflow