search

buildkite / test-collector-python

Python adapter for Buildkite Test Analytics
https://buildkite.com/test-analytics
MIT License
6 stars 5 forks source link

Reported test run durations are affected by freezegun #27

Closed gricey432 closed 2 months ago

gricey432 commented 4 months ago

Just tried this out for the first time and the reported runtimes for my suite as well as some of the tests are wildly wrong.

The suite ran in about 15 minutes.

2024-04-16_11-20-54

The tests which have the huge runtimes are all using Freezegun to mess with the clock https://github.com/spulec/freezegun which feels like it's probably related.

Freezegun has its own ignore system, unsure if that's the best fix for this or if the test collector should automatically handle libraries like this.

gricey432 commented 4 months ago

I've confirmed it's definitely an interaction with freezegun. Adding the following to conftest resolved it.

import freezegun

freezegun.configure(extend_ignore_list=['buildkite_test_collector'])
joshprice commented 2 months ago

Thanks for raising this, don't think we can reasonably support such behaviour automatically for all time mocking tools.

Going to close this issue based on the fact that the ignore list fix works around it.

See https://github.com/spulec/freezegun?tab=readme-ov-file#ignore-packages for more details.