search

rake-compiler / rake-compiler-dock

Easy to use and reliable cross compiler environment for building Windows, Linux, Mac and JRuby binary gems.
MIT License
78 stars 30 forks source link

Pass SOURCE_DATE_EPOCH to docker run #128

Closed segiddins closed 2 months ago

segiddins commented 3 months ago

See https://github.com/rubygems/rubygems/pull/2289, it helps make gem builds reproducible

segiddins commented 3 months ago

Ah I think build/sudoers also might need to be updated?

flavorjones commented 3 months ago

I've added a test and included it in the sudoers allowlist.

flavorjones commented 3 months ago

@segiddins Assuming this goes green, can you suggest some documentation improvements to help gem maintainers use this feature?

segiddins commented 3 months ago

Hmm. It's not something that's necessary for folks to use while building, but it is a pre-requisite to be able to rebuild a gem (because of timestamps in the tarball, which RubyGems will take from SOURCE_DATE_EPOCH, and also timestamps in the linked binary, and ld also respects SOURCE_DATE_EPOCH)

flavorjones commented 3 months ago

@segiddins Sorry, I'm not sure I understand your last response. Who is this feature for, what will they do with it, how will they use it? I think this is necessary both for user documentation but also so that I understand why this code is there as the maintainer.

ianks commented 2 months ago

@flavorjones This shouldn’t be a feature that users really need to know about, but it’s a key tenant of reproducible builds.. Certain OS, like nix, mandate its usage.

flavorjones commented 2 months ago

It still needs to be documented and I'd like at least for someone to verify that this works as expected.

flavorjones commented 2 months ago

My point here really is that we don't know if the feature actually works to generate reproducible gems. I wrote a test that the env var gets exported. And I know what the env var does. But does the whole stack work?

If nobody is willing to try it out and tell me it works, and explain how to build gems with it, it doesn't seem worth merging right now.

segiddins commented 2 months ago

Running CI=1 bundle exec rake gem:x86_64-linux-musl in test/rcd_test before this change twice (and mv pkg pkg-1 after the first invocation), and diffing the results

❯ diffoscope --max-diff-block-lines-saved 100 --exclude-directory-metadata yes pkg-1 pkg
--- pkg-1
+++ pkg
│   --- pkg-1/rcd_test-1.0.0-x86_64-linux-musl.gem
├── +++ pkg/rcd_test-1.0.0-x86_64-linux-musl.gem
│ ├── file list
│ │ @@ -1,3 +1,3 @@
│ │ --r--r--r--   0 wheel        (0) wheel        (0)      599 2024-09-24 15:44:57.000000 metadata.gz
│ │ --r--r--r--   0 wheel        (0) wheel        (0)     7298 2024-09-24 15:44:57.000000 data.tar.gz
│ │ --r--r--r--   0 wheel        (0) wheel        (0)      295 2024-09-24 15:44:57.000000 checksums.yaml.gz
│ │ +-r--r--r--   0 wheel        (0) wheel        (0)      599 2024-09-24 15:45:21.000000 metadata.gz
│ │ +-r--r--r--   0 wheel        (0) wheel        (0)     7295 2024-09-24 15:45:21.000000 data.tar.gz
│ │ +-r--r--r--   0 wheel        (0) wheel        (0)      297 2024-09-24 15:45:21.000000 checksums.yaml.gz
│ ├── metadata.gz
│ │ ├── filetype from file(1)
│ │ │ @@ -1 +1 @@
│ │ │ -gzip compressed data, last modified: Tue Sep 24 15:44:57 2024, max compression, from Unix
│ │ │ +gzip compressed data, last modified: Tue Sep 24 15:45:21 2024, max compression, from Unix
│ ├── data.tar.gz
│ │ ├── filetype from file(1)
│ │ │ @@ -1 +1 @@
│ │ │ -gzip compressed data, last modified: Tue Sep 24 15:44:57 2024, max compression, from Unix
│ │ │ +gzip compressed data, last modified: Tue Sep 24 15:45:21 2024, max compression, from Unix
│ │ ├── data.tar
│ │ │ ├── file list
│ │ │ │ @@ -1,14 +1,14 @@
│ │ │ │ --rw-r--r--   0 wheel        (0) wheel        (0)      569 2024-09-24 15:44:57.000000 ext/java/RcdTestExtService.java
│ │ │ │ --rw-r--r--   0 wheel        (0) wheel        (0)      528 2024-09-24 15:44:57.000000 ext/java/RubyRcdTest.java
│ │ │ │ --rw-r--r--   0 wheel        (0) wheel        (0)     4815 2024-09-24 15:44:57.000000 ext/mri/extconf.rb
│ │ │ │ --rw-r--r--   0 wheel        (0) wheel        (0)     1697 2024-09-24 15:44:57.000000 ext/mri/rcd_test_ext.c
│ │ │ │ --rw-r--r--   0 wheel        (0) wheel        (0)      216 2024-09-24 15:44:57.000000 ext/mri/rcd_test_ext.h
│ │ │ │ --rw-r--r--   0 wheel        (0) wheel        (0)      132 2024-09-24 15:44:57.000000 lib/rcd_test.rb
│ │ │ │ --rwxr-xr-x   0 wheel        (0) wheel        (0)     5920 2024-09-24 15:44:57.000000 lib/rcd_test/2.4/rcd_test_ext.so
│ │ │ │ --rwxr-xr-x   0 wheel        (0) wheel        (0)     5920 2024-09-24 15:44:57.000000 lib/rcd_test/2.5/rcd_test_ext.so
│ │ │ │ --rwxr-xr-x   0 wheel        (0) wheel        (0)     5920 2024-09-24 15:44:57.000000 lib/rcd_test/2.6/rcd_test_ext.so
│ │ │ │ --rwxr-xr-x   0 wheel        (0) wheel        (0)     5920 2024-09-24 15:44:57.000000 lib/rcd_test/2.7/rcd_test_ext.so
│ │ │ │ --rwxr-xr-x   0 wheel        (0) wheel        (0)     5920 2024-09-24 15:44:57.000000 lib/rcd_test/3.0/rcd_test_ext.so
│ │ │ │ --rwxr-xr-x   0 wheel        (0) wheel        (0)     5920 2024-09-24 15:44:57.000000 lib/rcd_test/3.1/rcd_test_ext.so
│ │ │ │ --rwxr-xr-x   0 wheel        (0) wheel        (0)     5920 2024-09-24 15:44:57.000000 lib/rcd_test/3.2/rcd_test_ext.so
│ │ │ │ --rwxr-xr-x   0 wheel        (0) wheel        (0)     5920 2024-09-24 15:44:57.000000 lib/rcd_test/3.3/rcd_test_ext.so
│ │ │ │ +-rw-r--r--   0 wheel        (0) wheel        (0)      569 2024-09-24 15:45:21.000000 ext/java/RcdTestExtService.java
│ │ │ │ +-rw-r--r--   0 wheel        (0) wheel        (0)      528 2024-09-24 15:45:21.000000 ext/java/RubyRcdTest.java
│ │ │ │ +-rw-r--r--   0 wheel        (0) wheel        (0)     4815 2024-09-24 15:45:21.000000 ext/mri/extconf.rb
│ │ │ │ +-rw-r--r--   0 wheel        (0) wheel        (0)     1697 2024-09-24 15:45:21.000000 ext/mri/rcd_test_ext.c
│ │ │ │ +-rw-r--r--   0 wheel        (0) wheel        (0)      216 2024-09-24 15:45:21.000000 ext/mri/rcd_test_ext.h
│ │ │ │ +-rw-r--r--   0 wheel        (0) wheel        (0)      132 2024-09-24 15:45:21.000000 lib/rcd_test.rb
│ │ │ │ +-rwxr-xr-x   0 wheel        (0) wheel        (0)     5920 2024-09-24 15:45:21.000000 lib/rcd_test/2.4/rcd_test_ext.so
│ │ │ │ +-rwxr-xr-x   0 wheel        (0) wheel        (0)     5920 2024-09-24 15:45:21.000000 lib/rcd_test/2.5/rcd_test_ext.so
│ │ │ │ +-rwxr-xr-x   0 wheel        (0) wheel        (0)     5920 2024-09-24 15:45:21.000000 lib/rcd_test/2.6/rcd_test_ext.so
│ │ │ │ +-rwxr-xr-x   0 wheel        (0) wheel        (0)     5920 2024-09-24 15:45:21.000000 lib/rcd_test/2.7/rcd_test_ext.so
│ │ │ │ +-rwxr-xr-x   0 wheel        (0) wheel        (0)     5920 2024-09-24 15:45:21.000000 lib/rcd_test/3.0/rcd_test_ext.so
│ │ │ │ +-rwxr-xr-x   0 wheel        (0) wheel        (0)     5920 2024-09-24 15:45:21.000000 lib/rcd_test/3.1/rcd_test_ext.so
│ │ │ │ +-rwxr-xr-x   0 wheel        (0) wheel        (0)     5920 2024-09-24 15:45:21.000000 lib/rcd_test/3.2/rcd_test_ext.so
│ │ │ │ +-rwxr-xr-x   0 wheel        (0) wheel        (0)     5920 2024-09-24 15:45:21.000000 lib/rcd_test/3.3/rcd_test_ext.so
│ ├── checksums.yaml.gz
│ │ ├── filetype from file(1)
│ │ │ @@ -1 +1 @@
│ │ │ -gzip compressed data, last modified: Tue Sep 24 15:44:57 2024, max compression, from Unix
│ │ │ +gzip compressed data, last modified: Tue Sep 24 15:45:21 2024, max compression, from Unix
│ │ ├── checksums.yaml
│ │ │ @@ -1,7 +1,7 @@
│ │ │  ---
│ │ │  SHA256:
│ │ │ -  metadata.gz: 745f6ecb74370f44bae4383c3c71e07d99802ff35f2a630fee912608bc5d60c9
│ │ │ -  data.tar.gz: c589c08c5cf9e42506681506453ee1632c1ace37a7957fd08b8077e76c0a27b6
│ │ │ +  metadata.gz: 8c5622a20327150fcf3f48cc7cc8e06c719bc93a365dfea2da5fc957dec12e1f
│ │ │ +  data.tar.gz: ab96733d2dcc104c1e8aa32a3702c756cf46b844a9f3231b25eade931ed38160
│ │ │  SHA512:
│ │ │ -  metadata.gz: 8cac7391d46ea0b4ee8a5ac99a0e52bc6d1c0ad8d5cb842f0ef5909c247172fad9b20063711fcc48f323e293e991d7b963f95cf0a1f54e63307669796f52f046
│ │ │ -  data.tar.gz: f87986e4ce87077a3a968388877fe8e10b47943750ecf312bb8a0c1296997846390c649ec299fd8deddb16b95448efd15e57667a9dbbca8fec2610116366c68d
│ │ │ +  metadata.gz: 37bf64cf1e7bc67c75787628e53abd78d19a7b672dddaef4aaf56c8aba5bf70192360ba33a1921cd68857722c69fdf81484dfb38f5dfbfbba864167225f2cfd2
│ │ │ +  data.tar.gz: e5a820b22e7c7f2afa910d82f9f6446f2fcba2004caaab0fd117ab641d7a3934ecf59606f8ab0bfb72d18daed296e9ff66aac5b06d7ac0dbdba00c3523a5af66

after

❯ sha256sum pkg*/*
sha256sum: pkg-1/rcd_test-1.0.0-x86_64-linux-musl: Is a directory
175acd6f5f345f11bd5b6ca071e728cf3ba8eb75333dfd18f94100bc11dda02f  pkg-1/rcd_test-1.0.0-x86_64-linux-musl.gem
sha256sum: pkg/rcd_test-1.0.0-x86_64-linux-musl: Is a directory
175acd6f5f345f11bd5b6ca071e728cf3ba8eb75333dfd18f94100bc11dda02f  pkg/rcd_test-1.0.0-x86_64-linux-musl.gem

So setting the source date epoch is enough (for a simple gem) to get a byte-for-byte identical build out of rake-compiler-dock. In normal use, you'd extract the SOURCE_DATE_EPOCH from the built gem (by looking at the timestamps in the .gem tar file), and then only set it on the rebuild

flavorjones commented 2 months ago

@segiddins Great! Thank you for exercising the stack.

I realize I must now be completely annoying you, but I feel you're missing an opportunity to educate people on what reproducible builds are and how to use this tool as part of that build chain by not adding something to the README. 🤷 But I'll stop asking and merge the feature, undocumented.

If you decide in the future that you want to add something to the README, even if it's a link pointing to something you (or someone else) has written, I would gladly merge it.

segiddins commented 2 months ago

I agree it's a great idea to document reproducible gem builds 😅 cc @duckinator, we should do that for the gem rebuild command you added

duckinator commented 2 months ago

Yeah, definitely.

It's not proper documentation, but there's a very rough WIP tutorial about it at https://pup-e.com/resources/reproducible-gem-builds/. It doesn't discuss the inner workings at all, just the user-facing parts, but it should be enough to try it out.