Open Nemo157 opened 8 months ago
That seems to be a lot of effort for a single test.
I think it would be easier to change the test to detect system support and bail out (e.g. by calling the mkdir
system utility). This pattern is already used in a bunch of tests.
The actual code change to support this is one line in libtest
, padded out to support being unstable in the linked implementation. It's not the only situation I've encountered this in, just the most recent one. I dislike having tests say they pass when they've actually not run, seeing the
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out; finished in $TIME
is a good reminder that you've got some tests not running on your system that you're relying on CI for.
I dislike having tests say they pass when they've actually not run,
Maybe we need additional capabilities in test return values? We can already return Result<(), E>
for pass/fail, but maybe a bool
or dedicated enum in the Ok
side could distinguish runtime-ignored tests.
Yeah, that was the alternative I put in which I think is the better solution for this particular example. I still feel that the ability to skip tests via an env-var is useful in other situations, is far easier to get stabilized, and is easier to work with while unstable (since it's local config, not a feature that needs to be added to the test code).
I do think an environment variable is reasonable anyway -- there's precedent in other test options like RUST_TEST_THREADS
and RUST_TEST_NOCAPTURE
.
Citing RUST_TEST_THREADS
as an example makes me wary because it has been misused / misunderstood.
Proposal
Problem statement
Sometimes it is not possible to run a specific test on a system because of environmental reasons. These can be skipped by using
cargo test -- --skip some_test_name
, but this must be specified every time tests are run.Motivating examples or use cases
cargo
has a testcargo_metadata_non_utf8
that tests it can be used while in a non-utf8 directory. That test is alreadycfg
gated because:But even this claim of working on linux is specific to some linux filesystems, attempting to run this test when the target-dir is on zfs with
utf8only=on
fails:This is a fundamental system restriction which means this test will never work here and should always be skipped.
Solution sketch
Extend
libtest
's--skip
CLI arg with aRUST_TEST_SKIP
environment variable, a comma separated list of filters that will be added on to those parsed from the args, allowing it to be permanently set in the working tree via utilities likedirenv
.Alternatives
Extend
libtest
's in-process API to allow this test to mark itself as unsupported when it gets this error code while setting up. I believe this would be a good solution to additionally have, and more useful in this specific case, but I think both have their uses and adding the environment variable is easier to get done first.Links and related work
https://github.com/rust-lang/rust/compare/master...Nemo157:rust:rust-test-skip-env-var
What happens now?
This issue contains an API change proposal (or ACP) and is part of the libs-api team feature lifecycle. Once this issue is filed, the libs-api team will review open proposals as capability becomes available. Current response times do not have a clear estimate, but may be up to several months.
Possible responses
The libs team may respond in various different ways. First, the team will consider the problem (this doesn't require any concrete solution or alternatives to have been proposed):
Second, if there's a concrete solution: