search

zopefoundation / Products.ZCatalog

Zope's indexing and search solution.
Other
5 stars 22 forks source link

`CompositeIndexPerformanceTest.testPerformance` not sufficiently robust #147

Closed d-maurer closed 1 year ago

d-maurer commented 1 year ago

Recently several Zope test runs failed due to the test above, e.g. "https://github.com/zopefoundation/Zope/actions/runs/5280695894/jobs/9553185163".

The test is very sensible to load conditions - conditions which are often not met on github. Spurious failures are the result causing overhead to check the causes for the test run failure.

It seems doubtful to test for the CompositeIndex performance every time the test suite runs. Performance typically depends on load conditions; its measurement requires great care to achieve a sufficient degree of robustness. Currently, the test above does not achieve this degree.

Please disable this test or make its activation dependent on an envvar.

d-maurer commented 1 year ago

The test above already provides a way to get disabled: it uses a pseudo layer (named PseudoLayer) for the sole purpose not to run when only unit tests are called for.

I will try to find a way to use this in the Zope tests definition to disable the test there.