CI: reprotest sometimes fails to set timestamp

[marmarek]

How to file a helpful issue

Qubes OS release

unrelated

Brief summary

Reprotest (package reproducibility test tool) sometimes fails to build the package at all due to timestamp setting issue

Steps to reproduce

Easiest to observe on core-agent-linux:

1. Open a PR to core-agent-linux
2. Observe repro:vm-bookworm job failing
3. If it worked, click restart to have it failed

Expected behavior

It works every time

Actual behavior

Sometimes fails with:

dpkg-source: informaci�n: extrayendo qubes-core-agent en build-experiment-1
dpkg-source: informaci�n: desempaquetando qubes-core-agent_4.2.14.orig.tar.xz
dpkg-source: informaci�n: desempaquetando qubes-core-agent_4.2.14-1+deb12u1+devel1.debian.tar.xz
dpkg-source: fallo: no se puede cambiar la marca de tiempo de �build-experiment-1/.pc/applied-patches�: Invalid argument
dpkg-source: informaci�n: using patch list from debian/patches/series

It could be related to some docker settings/permissions.

[marmarek]

This is still an issue on R4.2 tests too: https://gitlab.com/QubesOS/qubes-salt/-/jobs/5639283722#L2151

00:00:47,298 [executor:docker:bd4dbbb149] output: dpkg-source: avertissement: extraction d'un paquet source non sign.. (salt_3006.4-2+deb12u1+devel1.dsc)
00:00:47,809 [executor:docker:bd4dbbb149] output: dpkg-source: info: extraction de salt dans build-experiment-1
00:00:47,810 [executor:docker:bd4dbbb149] output: dpkg-source: info: extraction de salt_3006.4.orig.tar.gz
00:01:13,739 [executor:docker:bd4dbbb149] output: dpkg-source: info: extraction de salt_3006.4-2+deb12u1+devel1.debian.tar.xz
00:01:13,957 [executor:docker:bd4dbbb149] output: dpkg-source: erreur: impossible de modifier la date de build-experiment-1/.pc/applied-patches: Invalid argument

[rustybird]

Oops, I forgot about that. A draft comment from 7 weeks ago (feels more like 7 months):

"It's due to multiple bugs in recent versions of faketime. Here it's intercepting a call to the libc's utimes() and forwarding it with illegal values. Normally the interception is non-default behavior, but due to another bug it happens by default - unless the distribution (e.g. f38) has backported some changes from after the latest release, or FAKE_UTIME=y is passed (doing the opposite of what its name says), in which case" 😱 [paragraph ends here]

When a test run works even if though should be affected by the bugs, it seems to be because reprotest randomly chose to skip faketime.

Maybe CI should run reprotest with --vary=-time. (There's a proposal to make that the reprotest default.) Doesn't the time inherently "vary" a bit anyway even without faketime?

[marmarek]

Doesn't the time inherently "vary" a bit anyway even without faketime?

Hour/minutes/seconds - yes. Day - rarely. So, --vary=-time would hide issues like including just the build date somewhere in artifacts (like, date in the man page).

But maybe we can try FAKE_UTIME=y? Or backport faketime fix, at least into the CI container image?

[rustybird]

--vary=-time would hide issues like including just the build date somewhere in artifacts (like, date in the man page).

Ah. Too bad there's no Linux "clock namespace" that would allow actually changing the system time/date (without affecting anything else).

But maybe we can try FAKE_UTIME=y? Or backport faketime fix, at least into the CI container image?

IIRC setting FAKE_UTIME=y would either help or do nothing or trigger other bugs, depending on the faketime release and the distro patches applied on top. (The last unaffected faketime version seems to be v0.9.8. But after that - probably starting with d90c8c2 - it's like having to reverse engineer people's version control practices.)

[DemiMarie]

--vary=-time would hide issues like including just the build date somewhere in artifacts (like, date in the man page).

Ah. Too bad there's no Linux "clock namespace" that would allow actually changing the system time/date (without affecting anything else).

There are: man 7 time_namespaces.

[rustybird]

Yes but that only supports CLOCK_MONOTONIC and CLOCK_BOOTTIME. Wouldn't it have to do CLOCK_REALTIME to implement a faketime alternative?

[h01ger]

what reprotest version is used? maybe the time variation could be disabled for this package?

[rustybird]

Could the CI environment be configured to run reprotest in a separate VM (if it doesn't already)?

If so, here's a faketime in 10 lines of Bash. It's compatible with reprotest and uses a cutting-edge clock handling technique tentatively named "changing the global system clock, and then changing it back afterwards":

https://github.com/rustybird/realfaketime/blob/main/faketime

[rustybird]

@h01ger Seems like it would affect all packages and reprotest versions, and depend only on the libfaketime build and luck (did reprotest take the "chosen randomly not to fake" path; maybe also: did the dodgy values generated by libfaketime happen to fall into the valid range)

[h01ger]

In general I refrain from using faketime to fix reproducibility issues. For the same reasons I would also not recommend to use faketime to test for unreproducibility.

faketime is known to break some build tools.

[h01ger]

/me also recommends using dedicated systems running in a different time. It's fun to have machines in 2024 already! ;)

actually, they run in 2025 since a few hours:

$ date
Wed Jan  1 04:40:03 AM CET 2025

(that copy and paste is 2min old :)

[DemiMarie]

@rustybird I have no idea about using VMs. That’s @fepitre’s world, not mine. And yes, the lack of CLOCK_REALTIME support in time namespaces is a problem. That said, I think VMs are the only reasonable solution, unless one is willing to port faketime to use seccomp+ptrace. Otherwise programs that use raw system calls (such as ones written in Go) won’t work.

[marmarek]

Could the CI environment be configured to run reprotest in a separate VM (if it doesn't already)?

It is already. That VM then runs docker inside, I think mostly to ease software deployment, and AFAIR already is using a privileged container. But it is a single/fresh VM per reprotest job. The script is here: https://github.com/QubesOS/qubes-continuous-integration/blob/main/scripts/gitlab-reprotestv2