Open marmarek opened 1 year ago
This is still an issue on R4.2 tests too: https://gitlab.com/QubesOS/qubes-salt/-/jobs/5639283722#L2151
00:00:47,298 [executor:docker:bd4dbbb149] output: dpkg-source: avertissement: extraction d'un paquet source non sign.. (salt_3006.4-2+deb12u1+devel1.dsc)
00:00:47,809 [executor:docker:bd4dbbb149] output: dpkg-source: info: extraction de salt dans build-experiment-1
00:00:47,810 [executor:docker:bd4dbbb149] output: dpkg-source: info: extraction de salt_3006.4.orig.tar.gz
00:01:13,739 [executor:docker:bd4dbbb149] output: dpkg-source: info: extraction de salt_3006.4-2+deb12u1+devel1.debian.tar.xz
00:01:13,957 [executor:docker:bd4dbbb149] output: dpkg-source: erreur: impossible de modifier la date de build-experiment-1/.pc/applied-patches: Invalid argument
Oops, I forgot about that. A draft comment from 7 weeks ago (feels more like 7 months):
"It's due to multiple bugs in recent versions of faketime. Here it's intercepting a call to the libc's utimes()
and forwarding it with illegal values. Normally the interception is non-default behavior, but due to another bug it happens by default - unless the distribution (e.g. f38) has backported some changes from after the latest release, or FAKE_UTIME=y
is passed (doing the opposite of what its name says), in which case" 😱 [paragraph ends here]
When a test run works even if though should be affected by the bugs, it seems to be because reprotest randomly chose to skip faketime.
Maybe CI should run reprotest with --vary=-time
. (There's a proposal to make that the reprotest default.) Doesn't the time inherently "vary" a bit anyway even without faketime?
Doesn't the time inherently "vary" a bit anyway even without faketime?
Hour/minutes/seconds - yes. Day - rarely. So, --vary=-time
would hide issues like including just the build date somewhere in artifacts (like, date in the man page).
But maybe we can try FAKE_UTIME=y
? Or backport faketime fix, at least into the CI container image?
--vary=-time
would hide issues like including just the build date somewhere in artifacts (like, date in the man page).
Ah. Too bad there's no Linux "clock namespace" that would allow actually changing the system time/date (without affecting anything else).
But maybe we can try
FAKE_UTIME=y
? Or backport faketime fix, at least into the CI container image?
IIRC setting FAKE_UTIME=y
would either help or do nothing or trigger other bugs, depending on the faketime release and the distro patches applied on top. (The last unaffected faketime version seems to be v0.9.8. But after that - probably starting with d90c8c2 - it's like having to reverse engineer people's version control practices.)
--vary=-time
would hide issues like including just the build date somewhere in artifacts (like, date in the man page).Ah. Too bad there's no Linux "clock namespace" that would allow actually changing the system time/date (without affecting anything else).
There are: man 7 time_namespaces
.
Yes but that only supports CLOCK_MONOTONIC and CLOCK_BOOTTIME. Wouldn't it have to do CLOCK_REALTIME to implement a faketime alternative?
what reprotest version is used? maybe the time variation could be disabled for this package?
Could the CI environment be configured to run reprotest in a separate VM (if it doesn't already)?
If so, here's a faketime
in 10 lines of Bash. It's compatible with reprotest and uses a cutting-edge clock handling technique tentatively named "changing the global system clock, and then changing it back afterwards":
https://github.com/rustybird/realfaketime/blob/main/faketime
@h01ger Seems like it would affect all packages and reprotest versions, and depend only on the libfaketime build and luck (did reprotest take the "chosen randomly not to fake" path; maybe also: did the dodgy values generated by libfaketime happen to fall into the valid range)
In general I refrain from using faketime to fix reproducibility issues. For the same reasons I would also not recommend to use faketime to test for unreproducibility.
faketime is known to break some build tools.
/me also recommends using dedicated systems running in a different time. It's fun to have machines in 2024 already! ;)
actually, they run in 2025 since a few hours:
$ date Wed Jan 1 04:40:03 AM CET 2025
(that copy and paste is 2min old :)
@rustybird I have no idea about using VMs. That’s @fepitre’s world, not mine. And yes, the lack of CLOCK_REALTIME
support in time namespaces is a problem. That said, I think VMs are the only reasonable solution, unless one is willing to port faketime to use seccomp+ptrace. Otherwise programs that use raw system calls (such as ones written in Go) won’t work.
Could the CI environment be configured to run reprotest in a separate VM (if it doesn't already)?
It is already. That VM then runs docker inside, I think mostly to ease software deployment, and AFAIR already is using a privileged container. But it is a single/fresh VM per reprotest job. The script is here: https://github.com/QubesOS/qubes-continuous-integration/blob/main/scripts/gitlab-reprotestv2
How to file a helpful issue
Qubes OS release
unrelated
Brief summary
Reprotest (package reproducibility test tool) sometimes fails to build the package at all due to timestamp setting issue
Steps to reproduce
Easiest to observe on core-agent-linux:
Expected behavior
It works every time
Actual behavior
Sometimes fails with:
It could be related to some docker settings/permissions.