search

crate / crate-qa

CrateDB Quality Assurance
4 stars 1 forks source link

Timeouts in test_upgrade.StorageCompatibilityTest #305

Closed romseygeek closed 9 months ago

romseygeek commented 9 months ago

https://jenkins.crate.io/job/CrateDB/job/qa/job/crate_qa/763/execution/node/133/log/

This has been failing for a while now.

romseygeek commented 9 months ago

There is a seven-minute timeout on the _do_upgrade function in tests/bwc/test_upgrade.py, which tests a number of different version upgrades. The crate node logs reported to stdout start at 00:15:17 and finish at 00:22:34, which is over 7 minutes, so I think the issue here is just that as we've added more versions to test, the test is taking longer and we've finally hit the timeout.

Possible solutions: up the timeout, stop testing some of the older versions, split the test up so that the timeout is applied to individual upgrade tests rather than the whole suite.

mfussenegger commented 9 months ago

Nice. Let's try to increase it. I had somehow discarded that as a cause because locally a single upgrade cycle (going from 4.0 to latest) was fast enough to easily fit into the 7 minutes and the timeout is already split to each upgrade path. Should've checked the log timestamps.