search

sbt / sbt-native-packager

sbt Native Packager
https://sbt-native-packager.readthedocs.io/en/stable/
BSD 2-Clause "Simplified" License
1.59k stars 439 forks source link

Custom options for XZ Tarball not working #1426

Closed dEajL3kA closed 3 years ago

dEajL3kA commented 3 years ago

Expected behaviour

Accoridng to manual it should be possible to set the XZ options to be used by packageXzTarball via:

universalArchiveOptions in (Universal, packageXzTarball)

So I added this to my "build.sbt" file:

universalArchiveOptions in (Universal, packageXzTarball) := Seq("-3")

Expected behavior:

Intead of the packageXzTarball default XZ options, which are -9e (extremely slow!), it should now be using option -3.

Actual behaviour

It is easy to verify, e.g. via htop, that packageXzTarball still is using xz -9e, rather than the desired xz -3 😞

This completey locks up my system due to immense memory usage!

It appears that option universalArchiveOptions in (Universal, packageXzTarball) is ignored or documentation is wrong.

Or am I doing something wrong?

Information

muuki88 commented 3 years ago

Hi @dEajL3kA

Thanks for your issue. I can't reproduce this with simple build.sbt

enablePlugins(JavaAppPackaging)

Universal / packageXzTarball / universalArchiveOptions := Seq("-3")

The command line says:

Running with tar -3 /tmp/sbt_c1d3082d/my-app-0.1.0-SNAPSHOT.tar my-app-0.1.0-SNAPSHOT

Can you share an example project that fails for you?

dEajL3kA commented 3 years ago

I am working on a project based on Play Framework that uses SBT as build system.

In the build.sbt file we have:

universalArchiveOptions in (Universal, packageXzTarball) := Seq("-3")

Running ./sbt clean universal:packageXzTarball results and the .tar.xz package to be created. But it easy to see, e.g. via htop, that xz gets invoked with compression parameters -9e. Apperently the custom options are not used at all.

Note that xz with option -9e, the default used by sbt-native-packager, is extremly slow and use insane amount of memory!

(The build process failed for me several times because the OOM killer ended up killing my processes)

dEajL3kA commented 3 years ago

I am working on a project based on Play Framework that uses SBT as build system.

In the build.sbt file we have:

universalArchiveOptions in (Universal, packageXzTarball) := Seq("-3")

Running ./sbt clean universal:packageXzTarball results and the .tar.xz package to be created. But it easy to see, e.g. via htop, that xz gets invoked with compression parameters -9e. Apperently the custom options are not used at all.

Note that xz with option -9e, the default used by sbt-native-packager, is extremly slow and use insane amount of memory!

(The build process failed for me several times because the OOM killer ended up killing my processes)

Please see here:

sbt-universal-bug

muuki88 commented 3 years ago

I looked closer at the code and I wasn't aware the the "archive" and "compression" phase are separated.

The universalArchiveOptions are used for the tar command, but the following xz command is not configurable.

https://github.com/sbt/sbt-native-packager/blob/6583d703e986fcfc66f3cd7218bcaf3946a4bdb2/src/main/scala/com/typesafe/sbt/packager/universal/Archives.scala#L274

https://github.com/sbt/sbt-native-packager/blob/6583d703e986fcfc66f3cd7218bcaf3946a4bdb2/src/main/scala/com/typesafe/sbt/packager/universal/Archives.scala#L281

So what we actually need is a

  val universalCompressorOptions =
    SettingKey[Seq[String]]("universal-compressor-options", "Options passed to the compression command. Scope by task")

So that a user can configure

Universal / universalCompressionOptions / packageXzTarball := Seq("-3")
dEajL3kA commented 3 years ago

The universalArchiveOptions are used for the tar command, but the following xz command is not configurable.

I see. Makes a lot of sense why setting the xz options was not working then 😏

(This wasn't clear to me from the docs. Only by looking at the source codes I found that xz compressor options are hard-coded)

So what we actually need is a

val universalCompressorOptions =
 SettingKey[Seq[String]]("universal-compressor-options", "Options passed to the compression command. Scope by task")

So that a user can configure

Universal / universalCompressionOptions / packageXzTarball := Seq("-3")

Yeah, that would be much appreciated 🥳

(Even with this solution, I suggest changing the default from -9e to just -6, but definitely -e should be stripped)

Best regards.

muuki88 commented 3 years ago

Release 1.9.2 is on its way

dEajL3kA commented 3 years ago

Release 1.9.2 is on its way

Thanks! But how do I actually use that version in my project? https://stackoverflow.com/questions/68346952/how-to-use-sbt-native-packager-plug-in-v1-9-2-and-fix-not-found-error

SethTisue commented 3 years ago

1.9.2 isn't published for sbt 0.13. You'll need to upgrade to sbt 1.x. (1.5.5 just came out yesterday.)

dEajL3kA commented 3 years ago

1.9.2 isn't published for sbt 0.13. You'll need to upgrade to sbt 1.x. (1.5.5 just came out yesterday.)

Thanks you! I have now updated to sbt 1.x and to sbt-native-packager version 1.9.2.

Now it builds fine, but I still see the old behavior:

sbt-universal-xz-NEU

What am I missing? 😕

dEajL3kA commented 3 years ago

Never mind, after migrating to Play Framework 2.8.8 the sbt-native-packager v1.9.2 is working as expected now :partying_face:

(Still don't know why previous versions of Play Framework apprently used the old version, even though sbt-native-packager v1.9.2 was explicitely set in the plugins.sbt file. But, hopefully, I don't need to care anymore 😏)