NVIDIA / cuda-quantum

C++ and Python support for the CUDA Quantum programming model for heterogeneous quantum-classical workflows
https://nvidia.github.io/cuda-quantum/
Other
473 stars 172 forks source link

Make publishing tests run in repeatable order #2153

Closed bmhowe23 closed 2 weeks ago

bmhowe23 commented 3 weeks ago

Required prerequisites

Describe the bug

Our Publishing pipeline is not running tests in a repeatable order, which can cause non-repeatable behavior.

For example, one can compare the logs from the following runs

Image

The Testing photonics: line appears later in the other run.

Image

Non-repeatable behavior is just a nuisance sometimes, but it can also apparently cause some of our Python notebooks to fail if all the dependencies aren't correct.

The root cause is likely all of the find commands in validate_container.sh. E.g. https://github.com/NVIDIA/cuda-quantum/blob/735e62b27a2f534278b225e53453ad131b1cbf0b/scripts/validate_container.sh#L130. The find command returns results in a non-deterministic order (depending on how all the files were created), so its outputs likely need to be piped to sort to achieve a repeatable behavior.

Steps to reproduce the bug

See above.

Expected behavior

The publishing pipeline needs to run tests in repeatable order.

Is this a regression? If it is, put the last known working version (or commit) here.

Not a regression

Environment

Suggestions

No response

sacpis commented 2 weeks ago

"Non-repeatable behavior is just a nuisance sometimes, but it can also apparently cause some of our Python notebooks to fail if all the dependencies aren't correct."

Basically we just don't need sorting but sorting based on dependencies of these tests, right?