Open peisenha opened 4 years ago
Hi @peisenha,
I suggest trying to run it using python -m doctest -v temfpy/package/temfpy/uncertainty_quantification.py
and see if you have the same results. This will help narrow down if it is a doctest related problem or if it is related to pytest-stress.
I have looked at the code in question and indeed simple_linear_function
is defined in the same module, so things should just work... 🤔
Thanks for the swift response. Here is the output.
(base) peisenha@heracles:~/OpenSourceEconomics/temfpy/package$ python -m doctest -v temfpy/uncertainty_quantification.py
Trying:
x = [1, 2, 3, 4, 5, 6, 7, 8]
Expecting nothing
ok
Trying:
y = borehole(x)
Expecting nothing
ok
Trying:
np.testing.assert_almost_equal(y, 34.43500403827335)
Expecting nothing
ok
Trying:
x = [1, 2, 3]
Expecting nothing
ok
Trying:
y = eoq_model(x, r=0.1)
Expecting nothing
ok
Trying:
np.testing.assert_almost_equal(y, 18.973665961010276)
Expecting nothing
ok
Trying:
x = [1, 2, 3]
Expecting nothing
ok
Trying:
y = ishigami(x)
Expecting nothing
ok
Trying:
np.testing.assert_almost_equal(y, 10.037181146302519)
Expecting nothing
ok
Trying:
x = [1, 2, 3]
Expecting nothing
ok
Trying:
y = simple_linear_function(x)
Expecting nothing
ok
Trying:
np.testing.assert_almost_equal(y, 6)
Expecting nothing
ok
1 items had no tests:
uncertainty_quantification
4 items passed all tests:
3 tests in uncertainty_quantification.borehole
3 tests in uncertainty_quantification.eoq_model
3 tests in uncertainty_quantification.ishigami
3 tests in uncertainty_quantification.simple_linear_function
12 tests in 5 items.
12 passed and 0 failed.
Test passed.
Thanks! So indeed it seems a problem with pytest-stress
or pytest
.
Can you please uninstall pytest-stress
and execute pytest
again?
Also please post the full output you see in the terminal on failure.
Sorry for the delay. I am back at the keyboard and will respond quicker going forward now. Here you go ..
(base) peisenha@heracles:~/OpenSourceEconomics/temfpy/package$ py.test
Test session starts (platform: linux, Python 3.7.7, pytest 5.4.2, pytest-sugar 0.9.3)
Using --randomly-seed=1591724936
rootdir: /home/peisenha/external-storage/ownCloud/office/OpenSourceEconomics/temfpy/package, inifile: tox.ini
plugins: flakes-4.0.0, sugar-0.9.3, randomly-3.4.0, black-0.3.8, hypothesis-5.11.0, cov-2.8.1
collecting ...
temfpy/tests/test_uncertainty_quantification.py ✓✓✓✓✓✓ 100% ██████████
temfpy/tests/__init__.py ✓✓ 29% ██▉
temfpy/uncertainty_quantification.py ✓✓✓✓✓✓ 97% █████████▋
temfpy/optimization.py ✓✓✓✓ 90% █████████
docs/source/conf.py ✓✓ 71% ███████▏
temfpy/__init__.py ✓✓ 65% ██████▌
setup.py ✓✓ 94% █████████▍
bin/run-figures ✓ 39% ███▉
scripts/run_figures.py ✓✓ 74% ███████▌
docs/_static/codes/fig-eoq-tradeoff.py ✓✓ 84% ████████▍
temfpy/config.py ✓✓ 87% ████████▊
================================================================================================================================ warnings summary =================================================================================================================================
/home/peisenha/local/anaconda3/envs/temfpy/lib/python3.7/site-packages/pytest_flakes.py:51: 11 tests with warnings
/home/peisenha/local/anaconda3/envs/temfpy/lib/python3.7/site-packages/pytest_flakes.py:51: PytestDeprecationWarning: direct construction of FlakesItem has been deprecated, please use FlakesItem.from_parent
return FlakesItem(path, parent, flakes_ignore)
/home/peisenha/local/anaconda3/envs/temfpy/lib/python3.7/site-packages/pytest_black.py:26: 10 tests with warnings
/home/peisenha/local/anaconda3/envs/temfpy/lib/python3.7/site-packages/pytest_black.py:26: PytestDeprecationWarning: direct construction of BlackItem has been deprecated, please use BlackItem.from_parent
return BlackItem(path, parent)
temfpy/tests/test_uncertainty_quantification.py::test_simple_linear_function
/home/peisenha/external-storage/ownCloud/office/OpenSourceEconomics/temfpy/package/temfpy/uncertainty_quantification.py:200: RuntimeWarning: invalid value encountered in double_scalars
return sum(x)
temfpy/tests/test_uncertainty_quantification.py::test_borehole
/home/peisenha/external-storage/ownCloud/office/OpenSourceEconomics/temfpy/package/temfpy/uncertainty_quantification.py:60: RuntimeWarning: divide by zero encountered in double_scalars
rslt = a / (b * (1 + c / d + e))
temfpy/tests/test_uncertainty_quantification.py::test_borehole
/home/peisenha/external-storage/ownCloud/office/OpenSourceEconomics/temfpy/package/temfpy/uncertainty_quantification.py:60: RuntimeWarning: invalid value encountered in double_scalars
rslt = a / (b * (1 + c / d + e))
-- Docs: https://docs.pytest.org/en/latest/warnings.html
============================================================================================================================== Hypothesis Statistics ==============================================================================================================================
Results (2.48s):
31 passed
(base) peisenha@heracles:~/OpenSourceEconomics/temfpy/package$
So the run without pytest-stress
passes, indeed seems like a problem with pytest-stress
, thanks for confirming it.
I will defer now to the maintainers to investigate further. 😁
@nicoddemus Thanks for that initial troubleshooting!
@peisenha I'll check this out tonight after work. Thanks!
Thanks!
So I did some initial digging and found that:
FAILED temfpy/optimization.py::temfpy.optimization.ackley
FAILED temfpy/optimization.py::temfpy.optimization.rastrigin
FAILED temfpy/uncertainty_quantification.py::temfpy.uncertainty_quantification.borehole
FAILED temfpy/uncertainty_quantification.py::temfpy.uncertainty_quantification.eoq_model
FAILED temfpy/uncertainty_quantification.py::temfpy.uncertainty_quantification.ishigami
FAILED temfpy/uncertainty_quantification.py::temfpy.uncertainty_quantification.simple_linear_function
So I fixed it by disabling those 👆 tests. Just kidding!
temfpy/tests/test_uncertainty_quantification.py::test_eoq_model PASSED
temfpy/tests/test_uncertainty_quantification.py::test_ishigami PASSED
temfpy/tests/test_uncertainty_quantification.py::test_borehole PASSED
temfpy/tests/test_uncertainty_quantification.py::test_simple_linear_function PASSED
Seems like some state may not exist due to the looping nature of pytest-stress, compared to a fresh pytest session. I'm also not familiar with doctest to be honest.
I basically copied the logic from pytest's main.py and wrapped it in a while loop.
I'll do more of deep dive this weekend (my brain is too fried on the weekdays 🤯).
Thanks for your patience!
@peisenha I did some more digging.
It seems like doctest namespace gets cleared out after pytest-stress loop.
In the mean time, one quick hack you can do is explicitly add an import statement for the associated function in each doctest:
>>> from temfpy.optimization import ackley
>>> x = [0, 0]
>>> y = ackley(x)
>>> np.testing.assert_almost_equal(y, 0)
Is your use-case to run your tests for a certain amount of time or perhaps number of times (pytest-repeat)?
@nicoddemus Sorry to bug you, but do you have any thoughts on this? I'm happy to investigate further if you could point me in some direction (Pytest source code).
Thanks!
Dear Imran,
Thank you very much for your effort. Keep us posted! I am just integrating pytest-stress in this toy package first to learn about it more to then use it for our fleshed out research codes. There, with their long running test batteries, the features would be very useful. We have hacked sth. similar together which we are desperate to get rid of :)
Thanks again!
Philipp Eisenhauer Economist
Mail peisenha@protonmail.com Web www.eisenhauer.io Repository https://github.com/peisenha
Sent with ProtonMail Secure Email.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Monday, June 15, 2020 1:44 AM, Imran Mumtaz notifications@github.com wrote:
@peisenha I did some more digging.
It seems like doctest namespace gets cleared out after pytest-stress loop.
In the mean time, one quick hack you can do is explicitly add an import statement for the associated function in each doctest:
from
temfpy
.
optimization
import
ackley
x
=
[
0
,
0
]
y
=
ackley
(
x
)
np
.
testing
.
assert_almost_equal
(
y
,
0
)
Is your use-case to run your tests for a certain amount of time or perhaps number of times (pytest-repeat)?
@nicoddemus Sorry to bug you, but do you have any thoughts on this? I'm happy to investigate further if you could point me in some direction (Pytest source code).
Thanks!
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.
Hi @ImXron,
do you have any thoughts on this? I'm happy to investigate further if you could point me in some direction (Pytest source code).
pytest itself doesn't meddle with the doc test runner globals:
But doctest
itself will clear the namespace at the end of the test:
So your questions about the same test item being run more than once are pertinent, can @peisenha confirm it?
Yes, it passes the first time but then fails for the subsequent iterations.
Thank you very much for providing such a useful tool! I am trying to integrate it in the testing setup for our research codes at OpenSourceEconomics. However, I fail to get it to work in conjunction with
--doctest-modules
. It results in the following error:If I do not run the doctests, then all works as expected. Any guidance would be highly appreciated. Thanks again!