//tests:stamp_test inputs become stale too quickly

nacl commented 2 years ago

I often test all of rules_pkg using something like bazelisk test //....

When I leave my desk for the day and come back the following day, I sometimes see //tests:stamp_test mysteriously fail after making some changes. This is due to the timestamps no longer being valid:

Traceback (most recent call last):
  File "/private/var/tmp/_bazel_apsaltis/67ca5e5a723760c2131afe9d288ecae5/sandbox/darwin-sandbox/9/execroot/rules_pkg/bazel-out/darwin-fastbuild/bin/tests/stamp_test.runfiles/rules_pkg/tests/stamp_test.py", line 104, in test_not_epoch_times_tar
    self.assertTarFilesAreAlmostNew('stamped_tar.tar')
  File "/private/var/tmp/_bazel_apsaltis/67ca5e5a723760c2131afe9d288ecae5/sandbox/darwin-sandbox/9/execroot/rules_pkg/bazel-out/darwin-fastbuild/bin/tests/stamp_test.runfiles/rules_pkg/tests/stamp_test.py", line 82, in assertTarFilesAreAlmostNew
    self.check_mtime(info.mtime, file_path, info.name)
  File "/private/var/tmp/_bazel_apsaltis/67ca5e5a723760c2131afe9d288ecae5/sandbox/darwin-sandbox/9/execroot/rules_pkg/bazel-out/darwin-fastbuild/bin/tests/stamp_test.runfiles/rules_pkg/tests/stamp_test.py", line 63, in check_mtime
    self.fail(
AssertionError: Archive /private/var/tmp/_bazel_apsaltis/67ca5e5a723760c2131afe9d288ecae5/sandbox/darwin-sandbox/9/execroot/rules_pkg/bazel-out/darwin-fastbuild/bin/tests/stamp_test.runfiles/rules_pkg/tests/stamped_tar.tar contains file . with mtime:1639538959, expected:1639586337 +/- 10000

It would be nice if the relevant artifacts could be rebuilt when they have expired in the cache. I don't know how implement this, though.

This problem is never seen in CI, since it builds everything from scratch all the time.

aiuto commented 2 years ago

Yes. I see the same. We could up the window to make it happen every 2 or 3 days. I made my peace with doing a bazel --clean every so often. Even over multiple years, I think that will cost me far less time than trying to solve the problem.

nacl commented 2 years ago

Hm. That's fine, all things considered. Some options:

Fix this with documentation. The test failure output could be modified to say something about rerunning this test after removing the output.
Change the input to whatever timestamp the --stamp flag chooses to be something consistent and knowable in advance.
Dump the timestamp to a file somewhere and have the test read it to confirm the time.

Regardless, this is more of an annoyance than anything else. We can probably safely punt it to later on.

aiuto commented 2 years ago

I don't want to use a constant, because then I loose the check that I am actually getting the build time. This kind of integration tests against the real world, rather than a fixed one, does not play well into bazel's view of reproducibility. I addressed the rest in #497

achew22 commented 2 years ago

A possible way to address this would be to create a sh_test rule that has your distributed .bzl files as deps then sets up a WORKSPACE that loads them and invokes bazel build on the output. That would make the generation and validation steps invalidate in the same way at the same time. This is effectively what rules_go does with our integration tests which work... reasonably well.

There is also https://github.com/bazelbuild/bazel-integration-testing, which has a python launcher for achieving this. That repo is, unfortunately, long dormant and I handed over my OWNERS on it, but I know there has been some renewed interest from the SIG that's being spun up around rules since this is a recurring problem.

aiuto commented 2 years ago

That would work too. I've been trying to keep as much in unit tests as possible, but this is really an integration test that I have tried to squeeze into the unit test mindset.

We already have a few tests that create their own WORKSPACE and run bazel from within. It might be worthwhile to factor out the commonality so they are easier to set up.

On Fri, Dec 24, 2021 at 2:02 PM Andrew Z Allen @.***> wrote:

A possible way to address this would be to create a sh_test rule that has your distributed .bzl files as deps then sets up a WORKSPACE that loads them and invokes bazel build on the output. That would make the generation and validation steps invalidate in the same way at the same time. This is effectively what rules_go does with our integration tests https://github.com/bazelbuild/rules_go/tree/master/tests/integration which work... reasonably well.

There is also https://github.com/bazelbuild/bazel-integration-testing, which has a python launcher for achieving this. That repo is, unfortunately, long dormant and I handed over my OWNERS on it, but I know there has been some renewed interest from the SIG that's being spun up around rules since this is a recurring problem.

— Reply to this email directly, view it on GitHub https://github.com/bazelbuild/rules_pkg/issues/489#issuecomment-1000917383, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAXHHHC3OW52PJA5AT7P6TLUSS7UVANCNFSM5KED7NKA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>

bazelbuild / rules_pkg

//tests:stamp_test inputs become stale too quickly #489