Reproducibility support

p5pRT commented 6 years ago

Migrated from rt.perl.org#133452 (status was 'open')

Searchable as RT133452$

p5pRT commented 6 years ago

From nixos@crystalgamma.de

This is a bug report for perl from nixos@crystalgamma.de\, generated with the help of perlbug 1.40 running under perl 5.24.4.

As part of making the nixpkgs/NixOS bootstrapping process reproducible\, I am working on making the build of Perl deterministic. The upstream distribution of Perl however writes certain details about the configuration environment like the time or uname to files like config.h or perlbug.

This makes bitwise reproducibility difficult. It would be a useful for every distribution aiming for reproducibility if $cf_time were to default to the timestamp stored in SOURCE_DATE_EPOCH instead of the current time (see https://reproducible-builds.org/specs/source-date-epoch/ and https://reproducible-builds.org/docs/timestamps/ ). Furthermore\, it would be useful to have a more straightforward way to replace (or omit) the other configuration environment values (e. g. $cf_by\, $myuname).

Please find an example patch for the defaulting of $cf_time attached.

Flags: category=install severity=low

Site configuration information for perl 5.24.4:

Configured by user at Sat Apr 14 09:35:01 UTC 2018.

Summary of my perl5 (revision 5 version 24 subversion 4) configuration:
Platform: osname=linux\, osvers=4.16.13\, archname=ppc64le-linux-thread-multi uname='linux talos-nixos 4.16.13 #1-nixos smp wed may 30 06:17:45 utc 2018 ppc64le gnulinux ' config_args='-de -Dcc=cc -Uinstallusrbinperl -Dinstallstyle=lib/perl5 - Duseshrplib -Dlocincpth=/no-such-path/include -Dloclibpth=/nix/store/ bqz3x6hpbfxm6hmnnrkaz4bax8rna14b-glibc-2.27/lib -Dusethreads -Dprefix=/nix/ store/5vdbx07jr4v1dwdanwznnc37jcidmnjf-perl-5.24.4 -Dman1dir=/nix/store/ 5vdbx07jr4v1dwdanwznnc37jcidmnjf-perl-5.24.4/share/man/man1 -Dman3dir=/nix/ store/5vdbx07jr4v1dwdanwznnc37jcidmnjf-perl-5.24.4/share/man/man3' hint=recommended\, useposix=true\, d_sigaction=define useithreads=define\, usemultiplicity=define use64bitint=define\, use64bitall=define\, uselongdouble=undef usemymalloc=n\, bincompat5005=undef Compiler: cc='cc'\, ccflags ='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/no-such-path/include -D_LARGEFILE_SOURCE - D_FILE_OFFSET_BITS=64'\, optimize='-O1'\, cppflags='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe - fstack-protector-strong -I/no-such-path/include' ccversion=''\, gccversion='7.3.0'\, gccosandvers='' intsize=4\, longsize=8\, ptrsize=8\, doublesize=8\, byteorder=12345678\, doublekind=3 d_longlong=define\, longlongsize=8\, d_longdbl=define\, longdblsize=16\, longdblkind=7 ivtype='long'\, ivsize=8\, nvtype='double'\, nvsize=8\, Off_t='off_t'\, lseeksize=8 alignbytes=8\, prototype=define Linker and Libraries: ld='cc'\, ldflags =' -fstack-protector-strong -L/nix/store/ bqz3x6hpbfxm6hmnnrkaz4bax8rna14b-glibc-2.27/lib' libpth= libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc libc=\, so=so\, useshrplib=true\, libperl=libperl.so gnulibc_version='2.27' Dynamic Linking: dlsrc=dl_dlopen.xs\, dlext=so\, d_dlsymun=undef\, ccdlflags='-Wl\,-E -Wl\,- rpath\,/nix/store/5vdbx07jr4v1dwdanwznnc37jcidmnjf-perl-5.24.4/lib/ perl5/5.24.4/ppc64le-linux-thread-multi/CORE' cccdlflags='-fPIC'\, lddlflags='-shared -O1 -L/nix/store/ bqz3x6hpbfxm6hmnnrkaz4bax8rna14b-glibc-2.27/lib -fstack-protector-strong'

@INC for perl 5.24.4: /nix/store/5vdbx07jr4v1dwdanwznnc37jcidmnjf-perl-5.24.4/lib/perl5/ site_perl/5.24.4/ppc64le-linux-thread-multi /nix/store/5vdbx07jr4v1dwdanwznnc37jcidmnjf-perl-5.24.4/lib/perl5/ site_perl/5.24.4 /nix/store/5vdbx07jr4v1dwdanwznnc37jcidmnjf-perl-5.24.4/lib/perl5/5.24.4/ ppc64le-linux-thread-multi /nix/store/5vdbx07jr4v1dwdanwznnc37jcidmnjf-perl-5.24.4/lib/perl5/5.24.4

Environment for perl 5.24.4: HOME=/home/user LANG=en_US.UTF-8 LANGUAGE (unset) LD_LIBRARY_PATH (unset) LOGDIR (unset) PATH=/home/user/bin:/run/wrappers/bin:/home/user/.nix-profile/bin:/nix/ var/nix/profiles/default/bin:/run/current-system/sw/bin:/etc/profiles/per- user/user/bin PERL_BADLANG (unset) SHELL=/run/current-system/sw/bin/fish

p5pRT commented 6 years ago

From nixos@crystalgamma.de

cf_time.diff

```diff --- perl-5.24.3/Configure +++ perl-5.24.3/Configure @@ -3862,7 +3862,10 @@ . ./posthint.sh : who configured the system -cf_time=`LC_ALL=C; LANGUAGE=C; export LC_ALL; export LANGUAGE; $date 2>&1` +case "$SOURCE_DATE_EPOCH" in + "") cf_time=`LC_ALL=C; LANGUAGE=C; export LC_ALL; export LANGUAGE; $date 2>&1`;; + *) cf_time=`LC_ALL=C; LANGUAGE=C; export LC_ALL; export LANGUAGE; date -d "@$SOURCE_DATE_EPOCH" 2>&1` +esac case "$cf_by" in "") cf_by=`(logname) 2>/dev/null` ```

p5pRT commented 6 years ago

From @jkeenan

On Wed\, 15 Aug 2018 15:18:02 GMT\, nixos@crystalgamma.de wrote:

This is a bug report for perl from nixos@crystalgamma.de\, generated with the help of perlbug 1.40 running under perl 5.24.4.

----------------------------------------------------------------- As part of making the nixpkgs/NixOS bootstrapping process reproducible\, I am working on making the build of Perl deterministic. The upstream distribution of Perl however writes certain details about the configuration environment like the time or uname to files like config.h or perlbug.

This makes bitwise reproducibility difficult. It would be a useful for every distribution aiming for reproducibility if $cf_time were to default to the timestamp stored in SOURCE_DATE_EPOCH instead of the current time (see https://reproducible-builds.org/specs/source-date- epoch/ and https://reproducible-builds.org/docs/timestamps/ ). Furthermore\, it would be useful to have a more straightforward way to replace (or omit) the other configuration environment values (e. g. $cf_by\, $myuname).

Please find an example patch for the defaulting of $cf_time attached.

There was a little bit of discussion of the question of reproducible builds at last year's Perl 5 Core Hackathon:

https://github.com/p5h/2017/wiki/Proposed-Discussions#reproducible-builds

--- Flags: category=install severity=low --- Site configuration information for perl 5.24.4:

Configured by user at Sat Apr 14 09:35:01 UTC 2018.

Summary of my perl5 (revision 5 version 24 subversion 4) configuration:

Platform: osname=linux\, osvers=4.16.13\, archname=ppc64le-linux-thread-multi uname='linux talos-nixos 4.16.13 #1-nixos smp wed may 30 06:17:45 utc 2018 ppc64le gnulinux ' config_args='-de -Dcc=cc -Uinstallusrbinperl -Dinstallstyle=lib/perl5 - Duseshrplib -Dlocincpth=/no-such-path/include -Dloclibpth=/nix/store/ bqz3x6hpbfxm6hmnnrkaz4bax8rna14b-glibc-2.27/lib -Dusethreads -Dprefix=/nix/ store/5vdbx07jr4v1dwdanwznnc37jcidmnjf-perl-5.24.4 -Dman1dir=/nix/store/ 5vdbx07jr4v1dwdanwznnc37jcidmnjf-perl-5.24.4/share/man/man1 -Dman3dir=/nix/ store/5vdbx07jr4v1dwdanwznnc37jcidmnjf-perl-5.24.4/share/man/man3' hint=recommended\, useposix=true\, d_sigaction=define useithreads=define\, usemultiplicity=define use64bitint=define\, use64bitall=define\, uselongdouble=undef usemymalloc=n\, bincompat5005=undef Compiler: cc='cc'\, ccflags ='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict- aliasing -pipe -fstack-protector-strong -I/no-such-path/include -D_LARGEFILE_SOURCE - D_FILE_OFFSET_BITS=64'\, optimize='-O1'\, cppflags='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe - fstack-protector-strong -I/no-such-path/include' ccversion=''\, gccversion='7.3.0'\, gccosandvers='' intsize=4\, longsize=8\, ptrsize=8\, doublesize=8\, byteorder=12345678\, doublekind=3 d_longlong=define\, longlongsize=8\, d_longdbl=define\, longdblsize=16\, longdblkind=7 ivtype='long'\, ivsize=8\, nvtype='double'\, nvsize=8\, Off_t='off_t'\, lseeksize=8 alignbytes=8\, prototype=define Linker and Libraries: ld='cc'\, ldflags =' -fstack-protector-strong -L/nix/store/ bqz3x6hpbfxm6hmnnrkaz4bax8rna14b-glibc-2.27/lib' libpth= libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc libc=\, so=so\, useshrplib=true\, libperl=libperl.so gnulibc_version='2.27' Dynamic Linking: dlsrc=dl_dlopen.xs\, dlext=so\, d_dlsymun=undef\, ccdlflags='-Wl\,-E -Wl\,- rpath\,/nix/store/5vdbx07jr4v1dwdanwznnc37jcidmnjf-perl-5.24.4/lib/ perl5/5.24.4/ppc64le-linux-thread-multi/CORE' cccdlflags='-fPIC'\, lddlflags='-shared -O1 -L/nix/store/ bqz3x6hpbfxm6hmnnrkaz4bax8rna14b-glibc-2.27/lib -fstack-protector- strong'

--- @INC for perl 5.24.4: /nix/store/5vdbx07jr4v1dwdanwznnc37jcidmnjf-perl-5.24.4/lib/perl5/ site_perl/5.24.4/ppc64le-linux-thread-multi /nix/store/5vdbx07jr4v1dwdanwznnc37jcidmnjf-perl-5.24.4/lib/perl5/ site_perl/5.24.4 /nix/store/5vdbx07jr4v1dwdanwznnc37jcidmnjf-perl- 5.24.4/lib/perl5/5.24.4/ ppc64le-linux-thread-multi /nix/store/5vdbx07jr4v1dwdanwznnc37jcidmnjf-perl- 5.24.4/lib/perl5/5.24.4

--- Environment for perl 5.24.4: HOME=/home/user LANG=en_US.UTF-8 LANGUAGE (unset) LD_LIBRARY_PATH (unset) LOGDIR (unset) PATH=/home/user/bin:/run/wrappers/bin:/home/user/.nix- profile/bin:/nix/ var/nix/profiles/default/bin:/run/current- system/sw/bin:/etc/profiles/per- user/user/bin PERL_BADLANG (unset) SHELL=/run/current-system/sw/bin/fish

-- James E Keenan (jkeenan@cpan.org)

p5pRT commented 6 years ago

The RT System itself - Status changed from 'new' to 'open'

p5pRT commented 6 years ago

From @jkeenan

On Wed\, 15 Aug 2018 15:18:02 GMT\, nixos@crystalgamma.de wrote:

This is a bug report for perl from nixos@crystalgamma.de\, generated with the help of perlbug 1.40 running under perl 5.24.4.

----------------------------------------------------------------- As part of making the nixpkgs/NixOS bootstrapping process reproducible\, I am working on making the build of Perl deterministic. The upstream distribution of Perl however writes certain details about the configuration environment like the time or uname to files like config.h or perlbug.

This makes bitwise reproducibility difficult. It would be a useful for every distribution aiming for reproducibility if $cf_time were to default to the timestamp stored in SOURCE_DATE_EPOCH instead of the current time (see https://reproducible-builds.org/specs/source-date- epoch/ and https://reproducible-builds.org/docs/timestamps/ ). Furthermore\, it would be useful to have a more straightforward way to replace (or omit) the other configuration environment values (e. g. $cf_by\, $myuname).

Please find an example patch for the defaulting of $cf_time attached.

The patch was drawn against perl-5.24.3\, which is no longer supported. I did my best to apply it to perl 5 blead and created this branch for smoke-testing and further evaluation:

smoke-me/jkeenan/133452/cf_time-reproducible

Thank you very much.

-- James E Keenan (jkeenan@cpan.org)

p5pRT commented 6 years ago

From @doughera88

On Wed\, Aug 15\, 2018 at 08:18:02AM -0700\, (via RT) wrote:

# New Ticket Created by
# Please include the string: [perl #133452] # in the subject line of all future correspondence about this issue. # \<URL: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=133452 >

----------------------------------------------------------------- As part of making the nixpkgs/NixOS bootstrapping process reproducible\, I am working on making the build of Perl deterministic. The upstream distribution of Perl however writes certain details about the configuration environment like the time or uname to files like config.h or perlbug.

This makes bitwise reproducibility difficult.

I agree that wanting to make it easy to make reproducible builds is a reasonable goal.

It would be a useful for every distribution aiming for reproducibility if $cf_time were to default to the timestamp stored in SOURCE_DATE_EPOCH instead of the current time (see https://reproducible-builds.org/specs/source-date-epoch/ and https://reproducible-builds.org/docs/timestamps/ ). Furthermore\, it would be useful to have a more straightforward way to replace (or omit) the other configuration environment values (e. g. $cf_by\, $myuname).

cf_by can already be overridden on the Configure command line by specifying the -Dcf_by. I'm unclear what you mean by "a more straightforward way".

Please find an example patch for the defaulting of $cf_time attached.

+case "$SOURCE_DATE_EPOCH" in + "") cf_time=`LC_ALL=C; LANGUAGE=C; export LC_ALL; export LANGUAGE; $date 2>&1`;; + *) cf_time=`LC_ALL=C; LANGUAGE=C; export LC_ALL; export LANGUAGE; date -d "@$SOURCE_DATE_EPOCH" 2>&1` +esac

Unfortunately\, the use of date -d to print out an arbitrary date is not portable. In general\, I don't think you can count on the date command being able to output arbitrary dates. A simpler first step\, though\, would be to allow a Configure command-line override -Dcf_time. A trivial patch to do that is here:

Inline Patch

```diff diff --git a/Configure b/Configure index 0194d7f434..b93250f023 100755 --- a/Configure +++ b/Configure @@ -3864,8 +3864,13 @@ esac . ./posthint.sh +# Allow overrides for reproducible builds +case "$cf_time" in +"") + cf_time=`LC_ALL=C; LANGUAGE=C; export LC_ALL; export LANGUAGE; $date 2>&1` + ;; +esac : who configured the system -cf_time=`LC_ALL=C; LANGUAGE=C; export LC_ALL; export LANGUAGE; $date 2>&1` case "$cf_by" in "") cf_by=`(logname) 2>/dev/null` -- ```

Andy Dougherty doughera@lafayette.edu

p5pRT commented 6 years ago

From @jkeenan

On Fri\, 17 Aug 2018 20:49:48 GMT\, doughera wrote:

On Wed\, Aug 15\, 2018 at 08:18:02AM -0700\, (via RT) wrote:

# New Ticket Created by # Please include the string: [perl #133452] # in the subject line of all future correspondence about this issue. # \<URL: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=133452 >

----------------------------------------------------------------- As part of making the nixpkgs/NixOS bootstrapping process reproducible\, I am working on making the build of Perl deterministic. The upstream distribution of Perl however writes certain details about the configuration environment like the time or uname to files like config.h or perlbug.

This makes bitwise reproducibility difficult.

I agree that wanting to make it easy to make reproducible builds is a reasonable goal.

It would be a useful for every distribution aiming for reproducibility if $cf_time were to default to the timestamp stored in SOURCE_DATE_EPOCH instead of the current time (see https://reproducible-builds.org/specs/source-date- epoch/ and https://reproducible-builds.org/docs/timestamps/ ). Furthermore\, it would be useful to have a more straightforward way to replace (or omit) the other configuration environment values (e. g. $cf_by\, $myuname).

cf_by can already be overridden on the Configure command line by specifying the -Dcf_by. I'm unclear what you mean by "a more straightforward way".

Please find an example patch for the defaulting of $cf_time attached.

+case "$SOURCE_DATE_EPOCH" in + "") cf_time=`LC_ALL=C; LANGUAGE=C; export LC_ALL; export LANGUAGE; $date 2>&1`;; + *) cf_time=`LC_ALL=C; LANGUAGE=C; export LC_ALL; export LANGUAGE; date -d "@$SOURCE_DATE_EPOCH" 2>&1` +esac

Unfortunately\, the use of date -d to print out an arbitrary date is not portable. In general\, I don't think you can count on the date command being able to output arbitrary dates. A simpler first step\, though\, would be to allow a Configure command-line override -Dcf_time. A trivial patch to do that is here:

diff --git a/Configure b/Configure index 0194d7f434..b93250f023 100755 --- a/Configure +++ b/Configure @@ -3864\,8 +3864\,13 @@ esac

. ./posthint.sh

+# Allow overrides for reproducible builds +case "$cf_time" in +"") + cf_time=`LC_ALL=C; LANGUAGE=C; export LC_ALL; export LANGUAGE; $date 2>&1` + ;; +esac : who configured the system -cf_time=`LC_ALL=C; LANGUAGE=C; export LC_ALL; export LANGUAGE; $date 2>&1` case "$cf_by" in "") cf_by=`(logname) 2>/dev/null`

Branch for smoke-testing:

smoke-me/jkeenan/adougherty/133452/cf_time-reproducible

-- James E Keenan (jkeenan@cpan.org)

p5pRT commented 6 years ago

From nixos@crystalgamma.de

On Friday\, August 17\, 2018 10:49:48 PM CEST you wrote:

On Wed\, Aug 15\, 2018 at 08:18:02AM -0700\, (via RT) wrote:

# New Ticket Created by # Please include the string: [perl #133452] # in the subject line of all future correspondence about this issue. # \<URL: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=133452 >

----------------------------------------------------------------- As part of making the nixpkgs/NixOS bootstrapping process reproducible\, I am working on making the build of Perl deterministic. The upstream distribution of Perl however writes certain details about the configuration environment like the time or uname to files like config.h or perlbug.

This makes bitwise reproducibility difficult.

I agree that wanting to make it easy to make reproducible builds is a reasonable goal.

It would be a useful for every distribution aiming for reproducibility if $cf_time were to default to the timestamp stored in SOURCE_DATE_EPOCH instead of the current time (see https://reproducible-builds.org/specs/source-date-epoch/ and https://reproducible-builds.org/docs/timestamps/ ). Furthermore\, it would be useful to have a more straightforward way to replace (or omit) the other configuration environment values (e. g. $cf_by\, $myuname). cf_by can already be overridden on the Configure command line by specifying the -Dcf_by. I'm unclear what you mean by "a more straightforward way".

OK\, that is definitely useful. I don't know what I was thinking when I was asking for "a more straightforward way" … I'm kind of embarrased :(

Please find an example patch for the defaulting of $cf_time attached.

+case "$SOURCE_DATE_EPOCH" in + "") cf_time=`LC_ALL=C; LANGUAGE=C; export LC_ALL; export LANGUAGE; $date 2>&1`;; + *) cf_time=`LC_ALL=C; LANGUAGE=C; export LC_ALL; export LANGUAGE; date -d "@$SOURCE_DATE_EPOCH" 2>&1` +esac

Unfortunately\, the use of date -d to print out an arbitrary date is not portable. In general\, I don't think you can count on the date command being able to output arbitrary dates. A simpler first step\, though\, would be to allow a Configure command-line override -Dcf_time. A trivial patch to do that is here:

diff --git a/Configure b/Configure index 0194d7f434..b93250f023 100755 --- a/Configure +++ b/Configure @@ -3864\,8 +3864\,13 @@ esac

. ./posthint.sh

+# Allow overrides for reproducible builds +case "$cf_time" in +"") + cf_time=`LC_ALL=C; LANGUAGE=C; export LC_ALL; export LANGUAGE; $date 2>&1` + ;; +esac

: who configured the system

-cf_time=`LC_ALL=C; LANGUAGE=C; export LC_ALL; export LANGUAGE; $date 2>&1` case "$cf_by" in "") cf_by=`(logname) 2>/dev/null`

I feared as much. Being able to override cf_time sounds good enough though.

Perl / perl5