Bad test (or code): `test_getCatalogPlan_partial`

dataflake commented 1 year ago

The unit test test_getCatalogPlan_partial routinely fails on the macOS GitHub Actions runners when run as part of the Zope alltests test script, which tests Zope in conjunction with other important Zope packages, see traceback and GHA runner log output URL below. This test never fails for me on my local macOS setup. This leads me to believe that either the code being tested or the test itself can show race conditions in certain environments.

Failure in test test_getCatalogPlan_partial (Products.ZCatalog.tests.test_plan.TestCatalogPlan)
Traceback (most recent call last):
  File "/Users/runner/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/Users/runner/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/unittest/case.py", line 605, in run
    testMethod()
  File "/Users/runner/work/Zope/Zope/.tox/py36/eggs/Products.ZCatalog-6.3-py3.6.egg/Products/ZCatalog/tests/test_plan.py", line 313, in test_getCatalogPlan_partial
    cat.getCatalogPlan(query2).plan(), ["numbers", "num", "date"]
  File "/Users/runner/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/unittest/case.py", line 829, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/Users/runner/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/unittest/case.py", line 1028, in assertListEqual
    self.assertSequenceEqual(list1, list2, msg, seq_type=list)
  File "/Users/runner/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/unittest/case.py", line 1010, in assertSequenceEqual
    self.fail(msg)
  File "/Users/runner/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/unittest/case.py", line 670, in fail
    raise self.failureException(msg)
AssertionError: Lists differ: ['numbers', 'date', 'num'] != ['numbers', 'num', 'date']

First differing element 1:
'date'
'num'

- ['numbers', 'date', 'num']
+ ['numbers', 'num', 'date']

Example at https://github.com/zopefoundation/Zope/actions/runs/3479685094/jobs/5818540212

dataflake commented 1 year ago

Assigning @mamico - he added the test code.

mamico commented 1 year ago

@dataflake a quick test might be to increase the 'sleep' times here https://github.com/zopefoundation/Products.ZCatalog/blob/master/src/Products/ZCatalog/tests/test_plan.py#L247

... or something more cleaver to simulate a slow index

davisagli commented 1 year ago

My first guess here would not be a race condition, but a set getting converted to a list (sets do not keep their members in insertion order the way modern dicts do).

On Wed, Nov 16, 2022 at 6:24 AM Mauro Amico @.***> wrote:

@dataflake https://github.com/dataflake a quick test might be to increase the 'sleep' times here https://github.com/zopefoundation/Products.ZCatalog/blob/master/src/Products/ZCatalog/tests/test_plan.py#L247

... or something more cleaver to simulate a slow index

— Reply to this email directly, view it on GitHub https://github.com/zopefoundation/Products.ZCatalog/issues/140#issuecomment-1317103588, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAL65R3A5242WJEIVIJBMTWITVBTANCNFSM6AAAAAASCHSE7Q . You are receiving this because you are subscribed to this thread.Message ID: @.***>

davisagli commented 1 year ago

Oh, but this is a test of the query planner? Then yeah, something related to timing seems likely.

On Wed, Nov 16, 2022 at 6:56 AM David Glick @.***> wrote:

My first guess here would not be a race condition, but a set getting converted to a list (sets do not keep their members in insertion order the way modern dicts do).

On Wed, Nov 16, 2022 at 6:24 AM Mauro Amico @.***> wrote:

@dataflake https://github.com/dataflake a quick test might be to increase the 'sleep' times here https://github.com/zopefoundation/Products.ZCatalog/blob/master/src/Products/ZCatalog/tests/test_plan.py#L247

... or something more cleaver to simulate a slow index

— Reply to this email directly, view it on GitHub https://github.com/zopefoundation/Products.ZCatalog/issues/140#issuecomment-1317103588, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAL65R3A5242WJEIVIJBMTWITVBTANCNFSM6AAAAAASCHSE7Q . You are receiving this because you are subscribed to this thread.Message ID: @.***>

zopefoundation / Products.ZCatalog

Bad test (or code): `test_getCatalogPlan_partial` #140