Performance: many skips (with new nodeids, mostly spent at end of collection)

[blueyed]

Given t/t-many-skips.py it can be seen that testmon adds a lot of overhead:

import pytest

@pytest.mark.parametrize("a", range(0, 100))
def test(a):
    pytest.skip(str(a))

Without testmon:

% pytest t/t-many-skips.py
===================================== test session starts =====================================
platform linux -- Python 3.8.1, pytest-5.3.5.dev562+ga1aad030b.d20200223, py-1.8.2.dev3+g2b6bd292, pluggy-0.13.1
rootdir: …/src/pytest, inifile: tox.ini
plugins: …, testmon-1.0.2
collected 100 items

t/t-many-skips.py sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 69%]
sssssssssssssssssssssssssssssss                                                         [100%]

==================================== 100 skipped in 0.28s =====================================

With testmon:

% pytest --testmon t/t-many-skips.py
===================================== test session starts =====================================
platform linux -- Python 3.8.1, pytest-5.3.5.dev562+ga1aad030b.d20200223, py-1.8.2.dev3+g2b6bd292, pluggy-0.13.1
testmon: changed files: 60, skipping collection of 54 files, environment: default
rootdir: …/src/pytest, inifile: tox.ini
plugins: …, testmon-1.0.2
collected 100 items / 119 deselected

t/t-many-skips.py sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 69%]
sssssssssssssssssssssssssssssss                                                         [100%]

============================ 100 skipped, 119 deselected in 18.26s ============================

I think this could be used as a concrete example of trying to improve its performance.

Note that there is a huge delay with collection already. If I remember correctly this is caused by inserting many items separately into the SQLite DB (which should be done in a batch probably).) Changing the code (adding assert True to the beginning of the test, before the skipping) and re-running with testmon takes 5.56s - so ~2/3 of the time is spent during registering new nodeids apparently. pytest --testmon --collect-only with pytest's own tests takes ~45s! (.testmondata with a size of 516K)

[tarpas]

Daniel, I don't get such bad results. With your t-many-skips.py I get 0.99s vs 0.18s(no testmon).

When testmon uses it's data it's 0.02s.

pytest source code collection is 3.2s vs 6.3

testmon v1.0.3, pytest 6.0.1.

I hope it's something with your set-up. Thanks and let me know if you can reproduce it, or a way for me to reproduce.

[blueyed]

Have you checked pytest --testmon --co without .testmondata in a checkout of pytest? (it's the initial setup of the DB which is very slow in particular - looking at strace reveals that it's a lot of syncing of the SQLite DB apparently, i.e. there are likely a lot of single queries that all get committed etc?!)