search

bazelbuild / rules_pkg

Bazel rules for creating packages of many types (zip, tar, deb, rpm, ...)
Apache License 2.0
212 stars 166 forks source link

pkg_tar tgz hashes change with Fedora 40 and zlib-ng #866

Open ctrox opened 1 month ago

ctrox commented 1 month ago

We are using pkg_tar to package helm charts into tgz files (helm only supports .tgz/.tar.gz). I recently upgraded my system to Fedora 40 and noticed all the chart hashes changing. Turns out they switched from zlib to zlib-ng, which is API compatible but changes the resulting archives with the same input.

Is there some way to work around this? As far as I can see rules_pkg is just using python gzip which makes use of the OS-specific zlib library installed. This makes reproducible builds across different systems quite difficult (e.g. some team members use MacOS).

aiuto commented 1 month ago

You can pick your own zip. pkg_tar has a compressor attribute that lets you specify a binary which compresses a stream of data. Then you will need a version of zip that is compatible across all your build environments. You probably have to build that from scratch. You could make it a Bazel target, or build it and install it in your base build image. Then you need pkg_tar to do the right thing. Since you want compressor set globally for your organization, the best way is to vendor rules_pkg into your own third_party tree, then use it as a local repository from WORKSPACE or modlue.bazel. Change the default for compressor to point to your version.

Bonus points for figuring out a way to add a module extension to rules_pkg so that you can do this from the workspace without local modifications.

ctrox commented 1 month ago

Thanks for the quick reply, that should work. I have simply downgraded to zlib for the moment but will look into this when I have some time.

BoleynSu commented 1 month ago

Would it be possible to make it a toolchain so we do not need to patch rules_pkg?

aiuto commented 1 month ago

Sure, but that does not help alone. As I said above, you still need a module extension to make some choices globally for the workspace. Toolchaining it just adds one more complexity - you have to decide if you want to autodetect the zip tool or if you want to build your own, or if you want to point to the one which might not be in your $PATH.