search

pytest-dev / pytest-randomly

:game_die: Pytest plugin to randomly order tests and control random.seed
MIT License
631 stars 30 forks source link

Add tensorflow and pytorch random seed setting & Update requirements. #612

Open StarryZhang-whu opened 8 months ago

StarryZhang-whu commented 8 months ago

Tests and README are included.

Add support for setting tensorflow and pytorch random seed. It can help detect flaky tests due to the randomness, and provide reproducibility.

Reference: [1] Dutta, Saikat, et al. "Detecting flaky tests in probabilistic and machine learning applications." Proceedings of the 29th ACM SIGSOFT international symposium on software testing and analysis. 2020.

adamchainz commented 8 months ago

Why did you close this? I started reviewing and tidying last night. I think it is a great PR, thank you very much.

StarryZhang-whu commented 8 months ago

Why did you close this? I started reviewing and tidying last night. I think it is a great PR, thank you very much.

I tried adding pytorch and tensorflow requirements to pass the test, but it seems I did something wrong. This PR failed checks.

StarryZhang-whu commented 8 months ago

Should I resubmit this PR?

adamchainz commented 8 months ago

You didn’t mess anything up. The requirements are compiled on the OS you run requirements/compile.py, in your case macOS. Normally there aren’t any OS-specific packages, indeed across all my 33 open source projects there aren’t. But it seems that tensorflow is split into OS-specific packages, so the requirements currently have tensorflow-macos compiled in. Naturally, this doesn't install on the Linux CI runner.

I will try come up with a solution here, perhaps adding tensorflow to the .txt files without pinning the OS-specific packages.

StarryZhang-whu commented 8 months ago

Maybe you can use compile.py to regenerate these py3*.txt files as before, with I added tensorflow and pytorch in requirements.in.

Thanks for your explanation!

StarryZhang-whu commented 8 months ago

How is this PR going?

adamchainz commented 8 months ago

I will work on it when I can. Please do not ping. It is not helpful and you should have no expectation of free service. If you are going to post a message, provide information, such as "I tried using my fork in a project and it worked well" or similar.

adamchainz commented 8 months ago

Python 3.12 has a segmentation fault, both locally and on CI.

Traceback ``` Fatal Python error: Segmentation fault Current thread 0x00007f6e828aeb80 (most recent call first): Garbage-collecting File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/ast.py", line 52 in parse File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/_code/source.py", line 185 in getstatementrange_ast File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/_code/code.py", line 261 in getsource File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/_code/code.py", line 834 in _getentrysource File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/_code/code.py", line 932 in repr_traceback_entry File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/_code/code.py", line 994 in repr_traceback File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/_code/code.py", line 1064 in repr_excinfo File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/_code/code.py", line 699 in getrepr File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/nodes.py", line 464 in _repr_failure_py File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/nodes.py", line 541 in repr_failure File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/runner.py", line 409 in pytest_make_collect_report File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/pluggy/_callers.py", line 102 in _multicall File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/pluggy/_manager.py", line 119 in _hookexec File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/pluggy/_hooks.py", line 501 in __call__ File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/runner.py", line 565 in collect_one_node File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/main.py", line 839 in _collect_one_node File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/main.py", line 976 in genitems File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/main.py", line 981 in genitems File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/main.py", line 813 in perform_collect File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/main.py", line 349 in pytest_collection File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/pluggy/_callers.py", line 102 in _multicall File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/pluggy/_manager.py", line 119 in _hookexec File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/pluggy/_hooks.py", line 501 in __call__ File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/main.py", line 338 in _main File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/main.py", line 285 in wrap_session File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/main.py", line 332 in pytest_cmdline_main File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/pluggy/_callers.py", line 102 in _multicall File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/pluggy/_manager.py", line 119 in _hookexec File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/pluggy/_hooks.py", line 501 in __call__ File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/config/__init__.py", line 174 in main File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/config/__init__.py", line 197 in console_main File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/pytest/__main__.py", line 7 in File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/coverage/execfile.py", line 211 in run File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/coverage/cmdline.py", line 858 in do_run File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/coverage/cmdline.py", line 681 in command_line File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/coverage/cmdline.py", line 970 in main File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/coverage/__main__.py", line 10 in File "", line 88 in _run_code File "", line 198 in _run_module_as_main Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special (total: 20) ```

This occurs reliably and disappears when removing the tensorflow import. But it seems to be part of numpy. I have looked on both projects’ issue trackers and found nothing, and haven’t got the time to chase creating a minimal example. Would you be interested in doing that?

StarryZhang-whu commented 8 months ago

Yes, I'm trying to do so. But I'm not sure how to create a minimal example of this case.

I noticed that except Python 3.12, other versions of CI worked well. With the below dependencies installed:

py311: absl-py==2.1.0,asgiref==3.8.0,astunparse==1.6.3,certifi==2024.2.2,charset-normalizer==3.3.2,coverage==7.4.4,Django==5.0.3,execnet==2.0.2,factory-boy==3.3.0,Faker==24.3.0,filelock==3.13.1,flatbuffers==24.3.25,fsspec==2024.3.1,gast==0.5.4,google-pasta==0.2.0,grpcio==1.62.1,h5py==3.10.0,idna==3.6,iniconfig==2.0.0,Jinja2==3.1.3,keras==3.1.1,libclang==18.1.1,Markdown==3.6,markdown-it-py==3.0.0,MarkupSafe==2.1.5,mdurl==0.1.2,ml-dtypes==0.3.2,model-bakery==1.17.0,mpmath==1.3.0,namex==0.0.7,networkx==3.2.1,numpy==1.26.4,nvidia-cublas-cu12==12.1.3.1,nvidia-cuda-cupti-cu12==12.1.105,nvidia-cuda-nvrtc-cu12==12.1.105,nvidia-cuda-runtime-cu12==12.1.105,nvidia-cudnn-cu12==8.9.2.26,nvidia-cufft-cu12==11.0.2.54,nvidia-curand-cu12==10.3.2.106,nvidia-cusolver-cu12==11.4.5.107,nvidia-cusparse-cu12==12.1.0.106,nvidia-nccl-cu12==2.19.3,nvidia-nvjitlink-cu12==12.4.99,nvidia-nvtx-cu12==12.1.105,opt-einsum==3.3.0,optree==0.11.0,packaging==24.0,pip==24.0,pluggy==1.4.0,protobuf==4.25.3,Pygments==2.17.2,pytest==8.1.1,pytest-randomly @ file:///home/runner/work/pytest-randomly/pytest-randomly/.tox/.tmp/package/1/pytest_randomly-3.15.0-py3-none-any.whl#sha256=33ad92c347928f2b348b0575fe78fb9f5ee21fe8a97540399b386a28a2b4b17f,pytest-xdist==3.5.0,python-dateutil==2.9.0.post0,requests==2.31.0,rich==13.7.1,setuptools==69.1.0,six==1.16.0,sqlparse==0.4.4,sympy==1.12,tensorboard==2.16.2,tensorboard-data-server==0.7.2,tensorflow==2.16.1,tensorflow-io-gcs-filesystem==0.36.0,termcolor==2.4.0,torch==2.2.1,triton==2.2.0,typing_extensions==4.10.0,urllib3==2.2.1,Werkzeug==3.0.1,wheel==0.42.0,wrapt==1.16.0
py311: commands[0]> python -W error::ResourceWarning -W error::DeprecationWarning -W error::PendingDeprecationWarning -W 'ignore:pkg_resources is deprecated as an API:DeprecationWarning' -W 'ignore:datetime.datetime.utcfromtimestamp() is deprecated:DeprecationWarning' -m coverage run -m pytest -p no:randomly tests
============================= test session starts ==============================

While the Python 3.12 CI failed with the below dependencies installed:

py312: absl-py==2.1.0,asgiref==3.8.1,astunparse==1.6.3,certifi==2024.2.2,charset-normalizer==3.3.2,colorama==0.4.6,coverage==7.4.4,Django==5.0.3,execnet==2.0.2,factory-boy==3.3.0,Faker==24.4.0,filelock==3.13.3,flatbuffers==24.3.25,fsspec==2024.3.1,gast==0.5.4,google-pasta==0.2.0,grpcio==1.62.1,h5py==3.10.0,idna==3.6,iniconfig==2.0.0,Jinja2==3.1.3,keras==3.1.1,libclang==18.1.1,Markdown==3.6,markdown-it-py==3.0.0,MarkupSafe==2.1.5,mdurl==0.1.2,ml-dtypes==0.3.2,model-bakery==1.17.0,mpmath==1.3.0,namex==0.0.7,networkx==3.2.1,numpy==1.26.4,nvidia-cublas-cu12==12.1.3.1,nvidia-cuda-cupti-cu12==12.1.105,nvidia-cuda-nvrtc-cu12==12.1.105,nvidia-cuda-runtime-cu12==12.1.105,nvidia-cudnn-cu12==8.9.2.26,nvidia-cufft-cu12==11.0.2.54,nvidia-curand-cu12==10.3.2.106,nvidia-cusolver-cu12==11.4.5.107,nvidia-cusparse-cu12==12.1.0.106,nvidia-nccl-cu12==2.19.3,nvidia-nvjitlink-cu12==12.4.99,nvidia-nvtx-cu12==12.1.105,opt-einsum==3.3.0,optree==0.11.0,packaging==24.0,pip==24.0,pluggy==1.4.0,protobuf==4.25.3,Pygments==2.17.2,pytest==8.1.1,pytest-randomly @ file:///home/runner/work/pytest-randomly/pytest-randomly/.tox/.tmp/package/1/pytest_randomly-3.15.0-py3-none-any.whl#sha256=1480f46b08e60d5fc895996cda1e7d98ead282b7ea49cb95665a2f654e3e264d,pytest-xdist==3.5.0,python-dateutil==2.9.0.post0,requests==2.31.0,rich==13.7.1,setuptools==69.2.0,six==1.16.0,sqlparse==0.4.4,sympy==1.12,tensorboard==2.16.2,tensorboard-data-server==0.7.2,tensorflow==2.16.1,termcolor==2.4.0,torch==2.2.1,typing_extensions==4.10.0,tzdata==2024.1,urllib3==2.2.1,Werkzeug==3.0.1,wheel==0.43.0,wrapt==1.16.0
py312: commands[0]> python -W error::ResourceWarning -W error::DeprecationWarning -W error::PendingDeprecationWarning -W 'ignore:pkg_resources is deprecated as an API:DeprecationWarning' -W 'ignore:datetime.datetime.utcfromtimestamp() is deprecated:DeprecationWarning' -m coverage run -m pytest -p no:randomly tests
Fatal Python error: Segmentation fault

which tensorflow-io-gcs-filesystem package is missing. I guess there might be some compatibility issue between python and package. I will inquire about this issue in community and try to figure it out. Thanks for your help!

adamchainz commented 8 months ago

Huh, interesting. It would be surprising though if the missing package caused the segmentation fault. A segfault is when code reaches for memory outside of its allotted area.

adamchainz commented 6 months ago

Please don’t just close PRs when you give up. I may find the time to finish this, or another contributor may be interested.