sbt / sbt-assembly

Deploy über-JARs. Restart processes. (port of codahale/assembly-sbt)
MIT License
1.95k stars 224 forks source link

Add assemblyUnzipDirectory to AssemblyOption to use a different directory for unzipping jars #448

Open er1c opened 3 years ago

er1c commented 3 years ago

Adding as a draft for just reference for now

This is a draft implementation for https://github.com/sbt/sbt-assembly/issues/445

TODO

er1c commented 3 years ago

Found one issue with the hardlinks, one of my assumptions of the caches being on the same devices is flawed :)

[error] java.nio.file.FileSystemException: /local/data_importer/target/streams/_global/assembly/_global/streams/assembly/2669ae58f340f420d4802cc2289f5ba407c4e2bc_a9a7907ae54d79c51d9dd7e07bc71162aa0c71a2_0289763ca8ca20fbc99cf2a3bcea50af3cd6be74/META-INF/MANIFEST.MF -> /.cache/sbt/v1/assembly-dependencies/2669ae58f340f420d4802cc2289f5ba407c4e2bc_a9a7907ae54d79c51d9dd7e07bc71162aa0c71a2_0289763ca8ca20fbc99cf2a3bcea50af3cd6be74/META-INF/MANIFEST.MF: Invalid cross-device link
[error]     at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
[error]     at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
[error]     at sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:476)
[error]     at java.nio.file.Files.createLink(Files.java:1086)
[error]     at sbtassembly.AssemblyUtils$.copyFile(AssemblyUtils.scala:174)
[error]     at sbtassembly.AssemblyUtils$.copyImpl(AssemblyUtils.scala:152)
[error]     at sbtassembly.AssemblyUtils$.$anonfun$copy$1(AssemblyUtils.scala:138)
[error]     at scala.Function$.$anonfun$tupled$1(Function.scala:79)

This either needs to be caught and then just copied, or have a toggle earlier up to see if hardlink is supported

er1c commented 2 years ago

Got some corner case that I need to figure out:

[info] Copying shapeless_2.12-2.3.3.jar from unzip cache: /home/docker/.cache/sbt/v1/assembly/dependencies/91e75de52000c787900cf3b917ad2ffe0e5035ea_6041e2c4871650c556a9c6842e43c04ed462b11f_806e468a010b7087977fac295351e79a95d7c832, to: /shared/text_factor/scala/text_tokenizer/target/streams/_global/assembly/_global/streams/assembly/91e75de52000c787900cf3b917ad2ffe0e5035ea_6041e2c4871650c556a9c6842e43c04ed462b11f_806e468a010b7087977fac295351e79a95d7c832, for project: text-factor/text_factor
[info] Calculate mappings...
[info] Checking every *.class/*.jar file's SHA-1.
[info] Merging files...
[info] Strategy 'deduplicate' was applied to 6 files (Run the task at debug level to see details)
[info] Strategy 'discard' was applied to 417 files (Run the task at debug level to see details)
[info] Strategy 'filterDistinctLines' was applied to 6 files (Run the task at debug level to see details)
[info] Strategy 'rename' was applied to 6 files (Run the task at debug level to see details)
[info] SHA-1: d1fbb4a0902d74b3746362fdb849620f582b548e
[error] java.lang.IllegalArgumentException: requirement failed: Source file '/home/docker/.cache/sbt/v1/assembly/dependencies/346af639499c3d3ed6bb01314995d424e58af9b2_ac435f9d09ce25fc2c3dcade9b1a567466f6245a_806e468a010b7087977fac295351e79a95d7c832/META-INF/maven/com.amazonaws/aws-java-sdk-serverlessapplicationrepository/pom.properties' does not exist.
[error]     at scala.Predef$.require(Predef.scala:281)
[error]     at sbtassembly.AssemblyUtils$.copyFile(AssemblyUtils.scala:193)
[error]     at sbtassembly.AssemblyUtils$.copyImpl(AssemblyUtils.scala:178)
[error]     at sbtassembly.AssemblyUtils$.$anonfun$copy$1(AssemblyUtils.scala:164)
[error]     at scala.Function$.$anonfun$tupled$1(Function.scala:79)
[error]     at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
[error]     at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
[error]     at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
[error]     at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
[error]     at scala.collection.TraversableLike.map(TraversableLike.scala:286)
[error]     at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
[error]     at scala.collection.AbstractTraversable.map(Traversable.scala:108)
[error]     at sbtassembly.AssemblyUtils$.copy(AssemblyUtils.scala:164)
[error]     at sbtassembly.AssemblyUtils$.copyDirectory(AssemblyUtils.scala:152)
[error]     at sbtassembly.Assembly$.$anonfun$processDependencyJars$12(Assembly.scala:462)
[error]     at sbt.State$StateOpsImpl$$anon$1.call(State.scala:389)
[error]     at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:113)
[error]     at xsbt.boot.Locks$GlobalLock.withChannelRetries$1(Locks.scala:91)
[error]     at xsbt.boot.Locks$GlobalLock.$anonfun$withFileLock$1(Locks.scala:119)