search

con / duct

A helper to run a command, capture stdout/stderr and details about running
https://pypi.org/project/con-duct/
MIT License
3 stars 2 forks source link

speed up execution tests #46

Closed asmacdo closed 2 months ago

asmacdo commented 5 months ago

They aren't miserably slow, but we can tweak (and standardize) the time values in the test_execution args and probably make this blazing fast. Doesn't seem crucial to block release though.

duct pytest test/test_execution.py
duct is executing pytest test/test_execution.py...
Log files will be written to .duct/logs/2024.06.06T17.49.26-444723_
============================= test session starts ==============================
platform linux -- Python 3.11.5, pytest-7.4.4, pluggy-1.5.0
rootdir: /home/austin/devel/duct
configfile: tox.ini
plugins: cov-5.0.0
collected 8 items

test/test_execution.py ........                                          [100%]

============================== 8 passed in 4.20s ===============================

Exit Code: 0
Command: pytest test/test_execution.py
Log files location: .duct/logs/2024.06.06T17.49.26-444723_
Wall Clock Time: 4.352977275848389
Memory Peak Usage: 0.1%
CPU Peak Usage: 18.0%
 austin@fancy  ~/devel/duct   next ± duct pytest
duct is executing pytest...
Log files will be written to .duct/logs/2024.06.06T17.46.28-441631_
============================= test session starts ==============================
platform linux -- Python 3.11.5, pytest-7.4.4, pluggy-1.5.0
rootdir: /home/austin/devel/duct
configfile: tox.ini
plugins: cov-5.0.0
collected 38 items

test/test_execution.py ........                                          [ 21%]
test/test_helpers.py .....                                               [ 34%]
test/test_prepare_outputs.py ......                                      [ 50%]
test/test_report.py ....                                                 [ 60%]
test/test_tailpipe.py ...............                                    [100%]

============================== 38 passed in 5.71s ==============================

Exit Code: 0
Command: pytest
Log files location: .duct/logs/2024.06.06T17.46.28-441631_
Wall Clock Time: 5.862141370773315
Memory Peak Usage: 0.6%
CPU Peak Usage: 29.0%
austin@fancy  ~/devel/duct   next ± duct tox
duct is executing tox...
Log files will be written to .duct/logs/2024.06.06T17.46.41-442524_
.pkg: _optional_hooks> python /home/austin/miniconda3/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: get_requires_for_build_sdist> python /home/austin/miniconda3/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: get_requires_for_build_wheel> python /home/austin/miniconda3/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: prepare_metadata_for_build_wheel> python /home/austin/miniconda3/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: build_sdist> python /home/austin/miniconda3/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta
lint: install_package> python -I -m pip install --force-reinstall --no-deps /home/austin/devel/duct/.tox/.tmp/package/368/duct-0.0.1.tar.gz
lint: commands[0]> flake8 src test
lint: OK ✔ in 2.47 seconds
typing: install_package> python -I -m pip install --force-reinstall --no-deps /home/austin/devel/duct/.tox/.tmp/package/369/duct-0.0.1.tar.gz
typing: commands[0]> mypy src test
src/duct.py:59: error: Need type annotation for "max_values"  [var-annotated]
src/duct.py:61: error: Need type annotation for "_sample"  [var-annotated]
src/duct.py:64: error: Name "command" already defined on line 35  [no-redef]
src/duct.py:333: error: Incompatible types in assignment (expression has type "None", variable has type "TextIO | TailPipe | int")  [assignment]
src/duct.py:343: error: Incompatible types in assignment (expression has type "None", variable has type "TextIO | TailPipe | int")  [assignment]
Found 5 errors in 1 file (checked 8 source files)
typing: exit 1 (0.34 seconds) /home/austin/devel/duct> mypy src test pid=442917
typing: FAIL ✖ in 2.1 seconds
py38: skipped because could not find python interpreter with spec(s): py38
py38: SKIP ⚠ in 0.01 seconds
py39: skipped because could not find python interpreter with spec(s): py39
py39: SKIP ⚠ in 0.01 seconds
py310: skipped because could not find python interpreter with spec(s): py310
py310: SKIP ⚠ in 0.01 seconds
py311: install_package> python -I -m pip install --force-reinstall --no-deps /home/austin/devel/duct/.tox/.tmp/package/370/duct-0.0.1.tar.gz
py311: commands[0]> pytest test
============================= test session starts ==============================
platform linux -- Python 3.11.5, pytest-8.2.0, pluggy-1.5.0
cachedir: .tox/py311/.pytest_cache
rootdir: /home/austin/devel/duct
configfile: tox.ini
plugins: mock-3.14.0, cov-5.0.0
collected 38 items

test/test_execution.py ........                                          [ 21%]
test/test_helpers.py .....                                               [ 34%]
test/test_prepare_outputs.py ......                                      [ 50%]
test/test_report.py ....                                                 [ 60%]
test/test_tailpipe.py ...............                                    [100%]

============================== 38 passed in 5.55s ==============================
py311: OK ✔ in 7.42 seconds
py312: skipped because could not find python interpreter with spec(s): py312
py312: SKIP ⚠ in 0.01 seconds
pypy3: skipped because could not find python interpreter with spec(s): pypy3
  lint: OK (2.47=setup[2.32]+cmd[0.15] seconds)
  typing: FAIL code 1 (2.10=setup[1.76]+cmd[0.34] seconds)
  py38: SKIP (0.01 seconds)
  py39: SKIP (0.01 seconds)
  py310: SKIP (0.01 seconds)
  py311: OK (7.42=setup[1.71]+cmd[5.71] seconds)
  py312: SKIP (0.01 seconds)
  pypy3: SKIP (0.01 seconds)
  evaluation failed :( (12.06 seconds)

Exit Code: 255
Command: tox
Log files location: .duct/logs/2024.06.06T17.46.41-442524_
Wall Clock Time: 12.165053367614746
Memory Peak Usage: 0.7%
CPU Peak Usage: 89.5%
yarikoptic commented 3 months ago

it is worthwhile, meanwhile can use pytest-xdist and parallelize to e.g. 5 to gain some speedup. E.g.

❯ time python -m pytest test
================================== test session starts ==================================
platform linux -- Python 3.12.4, pytest-8.2.0, pluggy-1.5.0
rootdir: /home/yoh/proj/CON/duct
configfile: tox.ini
plugins: mock-3.14.0, cov-5.0.0, xdist-3.6.1
collected 68 items                                                                      

test/test_execution.py ............                                               [ 17%]
test/test_log_paths.py ..............                                             [ 38%]
test/test_prepare_outputs.py ................                                     [ 61%]
test/test_report.py ........                                                      [ 73%]
test/test_tailpipe.py ...............                                             [ 95%]
test/test_validation.py ...                                                       [100%]

---------- coverage: platform linux, python 3.12.4-final-0 -----------
Name                       Stmts   Miss Branch BrPart   Cover   Missing
-----------------------------------------------------------------------
src/con_duct/__init__.py       1      0      0      0 100.00%
src/con_duct/__main__.py     344     17    104      7  94.20%   51, 66, 279->exit, 297-298, 315->314, 339-340, 545, 581-582, 599-600, 654-655, 689-690, 714, 718->722, 727->735, 743
-----------------------------------------------------------------------
TOTAL                        345     17    104      7  94.21%

================================== 68 passed in 15.71s ==================================
python -m pytest test  14.55s user 5.83s system 125% cpu 16.258 total
❯ time python -m pytest -n 5 test
================================== test session starts ==================================
platform linux -- Python 3.12.4, pytest-8.2.0, pluggy-1.5.0
rootdir: /home/yoh/proj/CON/duct
configfile: tox.ini
plugins: mock-3.14.0, cov-5.0.0, xdist-3.6.1
5 workers [68 items]    
....................................................................              [100%]

---------- coverage: platform linux, python 3.12.4-final-0 -----------
Name                       Stmts   Miss Branch BrPart   Cover   Missing
-----------------------------------------------------------------------
src/con_duct/__init__.py       1      0      0      0 100.00%
src/con_duct/__main__.py     344     17    104      7  94.20%   51, 66, 279->exit, 297-298, 315->314, 339-340, 545, 581-582, 599-600, 654-655, 689-690, 714, 718->722, 727->735, 743
-----------------------------------------------------------------------
TOTAL                        345     17    104      7  94.21%

================================== 68 passed in 9.34s ===================================
python -m pytest -n 5 test  24.21s user 6.65s system 310% cpu 9.936 total

so still 9 seconds but faster than original 15.

yarikoptic commented 2 months ago

I doubt it is worth spending time on ATM -- they are speedy enough for now, and some recent changes might help to reduce it further.