Open peisenha opened 4 years ago
nicoddemus
commented
4 years ago Hi @peisenha,
I suggest trying to run it using python -m doctest -v temfpy/package/temfpy/uncertainty_quantification.py and see if you have the same results. This will help narrow down if it is a doctest related problem or if it is related to pytest-stress.
I have looked at the code in question and indeed simple_linear_function is defined in the same module, so things should just work... 🤔
peisenha
commented
4 years ago Thanks for the swift response. Here is the output.
(base) peisenha@heracles:~/OpenSourceEconomics/temfpy/package$ python -m doctest -v temfpy/uncertainty_quantification.py
Trying:
x = [1, 2, 3, 4, 5, 6, 7, 8]
Expecting nothing
ok
Trying:
y = borehole(x)
Expecting nothing
ok
Trying:
np.testing.assert_almost_equal(y, 34.43500403827335)
Expecting nothing
ok
Trying:
x = [1, 2, 3]
Expecting nothing
ok
Trying:
y = eoq_model(x, r=0.1)
Expecting nothing
ok
Trying:
np.testing.assert_almost_equal(y, 18.973665961010276)
Expecting nothing
ok
Trying:
x = [1, 2, 3]
Expecting nothing
ok
Trying:
y = ishigami(x)
Expecting nothing
ok
Trying:
np.testing.assert_almost_equal(y, 10.037181146302519)
Expecting nothing
ok
Trying:
x = [1, 2, 3]
Expecting nothing
ok
Trying:
y = simple_linear_function(x)
Expecting nothing
ok
Trying:
np.testing.assert_almost_equal(y, 6)
Expecting nothing
ok
1 items had no tests:
uncertainty_quantification
4 items passed all tests:
3 tests in uncertainty_quantification.borehole
3 tests in uncertainty_quantification.eoq_model
3 tests in uncertainty_quantification.ishigami
3 tests in uncertainty_quantification.simple_linear_function
12 tests in 5 items.
12 passed and 0 failed.
Test passed.
Thanks! So indeed it seems a problem with pytest-stress or pytest.
Can you please uninstall pytest-stress and execute pytest again?
Also please post the full output you see in the terminal on failure.
peisenha
commented
4 years ago Sorry for the delay. I am back at the keyboard and will respond quicker going forward now. Here you go ..
(base) peisenha@heracles:~/OpenSourceEconomics/temfpy/package$ py.test
Test session starts (platform: linux, Python 3.7.7, pytest 5.4.2, pytest-sugar 0.9.3)
Using --randomly-seed=1591724936
rootdir: /home/peisenha/external-storage/ownCloud/office/OpenSourceEconomics/temfpy/package, inifile: tox.ini
plugins: flakes-4.0.0, sugar-0.9.3, randomly-3.4.0, black-0.3.8, hypothesis-5.11.0, cov-2.8.1
collecting ...
temfpy/tests/test_uncertainty_quantification.py ✓✓✓✓✓✓ 100% ██████████
temfpy/tests/__init__.py ✓✓ 29% ██▉
temfpy/uncertainty_quantification.py ✓✓✓✓✓✓ 97% █████████▋
temfpy/optimization.py ✓✓✓✓ 90% █████████
docs/source/conf.py ✓✓ 71% ███████▏
temfpy/__init__.py ✓✓ 65% ██████▌
setup.py ✓✓ 94% █████████▍
bin/run-figures ✓ 39% ███▉
scripts/run_figures.py ✓✓ 74% ███████▌
docs/_static/codes/fig-eoq-tradeoff.py ✓✓ 84% ████████▍
temfpy/config.py ✓✓ 87% ████████▊
================================================================================================================================ warnings summary =================================================================================================================================
/home/peisenha/local/anaconda3/envs/temfpy/lib/python3.7/site-packages/pytest_flakes.py:51: 11 tests with warnings
/home/peisenha/local/anaconda3/envs/temfpy/lib/python3.7/site-packages/pytest_flakes.py:51: PytestDeprecationWarning: direct construction of FlakesItem has been deprecated, please use FlakesItem.from_parent
return FlakesItem(path, parent, flakes_ignore)
/home/peisenha/local/anaconda3/envs/temfpy/lib/python3.7/site-packages/pytest_black.py:26: 10 tests with warnings
/home/peisenha/local/anaconda3/envs/temfpy/lib/python3.7/site-packages/pytest_black.py:26: PytestDeprecationWarning: direct construction of BlackItem has been deprecated, please use BlackItem.from_parent
return BlackItem(path, parent)
temfpy/tests/test_uncertainty_quantification.py::test_simple_linear_function
/home/peisenha/external-storage/ownCloud/office/OpenSourceEconomics/temfpy/package/temfpy/uncertainty_quantification.py:200: RuntimeWarning: invalid value encountered in double_scalars
return sum(x)
temfpy/tests/test_uncertainty_quantification.py::test_borehole
/home/peisenha/external-storage/ownCloud/office/OpenSourceEconomics/temfpy/package/temfpy/uncertainty_quantification.py:60: RuntimeWarning: divide by zero encountered in double_scalars
rslt = a / (b * (1 + c / d + e))
temfpy/tests/test_uncertainty_quantification.py::test_borehole
/home/peisenha/external-storage/ownCloud/office/OpenSourceEconomics/temfpy/package/temfpy/uncertainty_quantification.py:60: RuntimeWarning: invalid value encountered in double_scalars
rslt = a / (b * (1 + c / d + e))
-- Docs: https://docs.pytest.org/en/latest/warnings.html
============================================================================================================================== Hypothesis Statistics ==============================================================================================================================
Results (2.48s):
31 passed
(base) peisenha@heracles:~/OpenSourceEconomics/temfpy/package$ So the run without pytest-stress passes, indeed seems like a problem with pytest-stress, thanks for confirming it.
I will defer now to the maintainers to investigate further. 😁
ImXron
commented
4 years ago @nicoddemus Thanks for that initial troubleshooting!
@peisenha I'll check this out tonight after work. Thanks!
peisenha
commented
4 years ago Thanks!
ImXron
commented
4 years ago So I did some initial digging and found that:
FAILED temfpy/optimization.py::temfpy.optimization.ackley
FAILED temfpy/optimization.py::temfpy.optimization.rastrigin
FAILED temfpy/uncertainty_quantification.py::temfpy.uncertainty_quantification.borehole
FAILED temfpy/uncertainty_quantification.py::temfpy.uncertainty_quantification.eoq_model
FAILED temfpy/uncertainty_quantification.py::temfpy.uncertainty_quantification.ishigami
FAILED temfpy/uncertainty_quantification.py::temfpy.uncertainty_quantification.simple_linear_functionSo I fixed it by disabling those 👆 tests. Just kidding!
temfpy/tests/test_uncertainty_quantification.py::test_eoq_model PASSED
temfpy/tests/test_uncertainty_quantification.py::test_ishigami PASSED
temfpy/tests/test_uncertainty_quantification.py::test_borehole PASSED
temfpy/tests/test_uncertainty_quantification.py::test_simple_linear_function PASSEDSeems like some state may not exist due to the looping nature of pytest-stress, compared to a fresh pytest session. I'm also not familiar with doctest to be honest.
I basically copied the logic from pytest's main.py and wrapped it in a while loop.
I'll do more of deep dive this weekend (my brain is too fried on the weekdays 🤯).
Thanks for your patience!
ImXron
commented
4 years ago @peisenha I did some more digging.
It seems like doctest namespace gets cleared out after pytest-stress loop.
In the mean time, one quick hack you can do is explicitly add an import statement for the associated function in each doctest:
>>> from temfpy.optimization import ackley
>>> x = [0, 0]
>>> y = ackley(x)
>>> np.testing.assert_almost_equal(y, 0)Is your use-case to run your tests for a certain amount of time or perhaps number of times (pytest-repeat)?
@nicoddemus Sorry to bug you, but do you have any thoughts on this? I'm happy to investigate further if you could point me in some direction (Pytest source code).
Thanks!
peisenha
commented
4 years ago Dear Imran,
Thank you very much for your effort. Keep us posted! I am just integrating pytest-stress in this toy package first to learn about it more to then use it for our fleshed out research codes. There, with their long running test batteries, the features would be very useful. We have hacked sth. similar together which we are desperate to get rid of :)
Thanks again!
Philipp Eisenhauer Economist
Mail peisenha@protonmail.com Web www.eisenhauer.io Repository https://github.com/peisenha
Sent with ProtonMail Secure Email.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Monday, June 15, 2020 1:44 AM, Imran Mumtaz notifications@github.com wrote:
@peisenha I did some more digging.
It seems like doctest namespace gets cleared out after pytest-stress loop.
In the mean time, one quick hack you can do is explicitly add an import statement for the associated function in each doctest:
from
temfpy
.
optimization
import
ackley
x
=
[
0
,
0
]
y
=
ackley
(
x
)
np
.
testing
.
assert_almost_equal
(
y
,
0
)
Is your use-case to run your tests for a certain amount of time or perhaps number of times (pytest-repeat)?
@nicoddemus Sorry to bug you, but do you have any thoughts on this? I'm happy to investigate further if you could point me in some direction (Pytest source code).
Thanks!
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.
nicoddemus
commented
4 years ago Hi @ImXron,
do you have any thoughts on this? I'm happy to investigate further if you could point me in some direction (Pytest source code).
pytest itself doesn't meddle with the doc test runner globals:
But doctest itself will clear the namespace at the end of the test:
So your questions about the same test item being run more than once are pertinent, can @peisenha confirm it?
peisenha
commented
4 years ago Yes, it passes the first time but then fails for the subsequent iterations.
Thank you very much for providing such a useful tool! I am trying to integrate it in the testing setup for our research codes at OpenSourceEconomics. However, I fail to get it to work in conjunction with
--doctest-modules. It results in the following error:If I do not run the doctests, then all works as expected. Any guidance would be highly appreciated. Thanks again!