search

ramonhagenaars / nptyping

💡 Type hints for Numpy and Pandas
MIT License
576 stars 29 forks source link

test_instance_check_performance fails on some architectures #73

Open mcepl opened 2 years ago

mcepl commented 2 years ago

When packaging this package for openSUSE I have hit the similar problem on two non-x86_64 architectures with the test test_instance_check_performance. On i586 I get:

[  168s] ======================================================================
[  168s] FAIL: test_instance_check_performance (tests.test_performance.PerformanceTest)
[  168s] ----------------------------------------------------------------------
[  168s] Traceback (most recent call last):
[  168s]   File "/home/abuild/rpmbuild/BUILD/nptyping-2.0.1/tests/test_performance.py", line 25, in test_instance_check_performance
[  168s]     self.assertLess(second_time_sec, first_time_sec)
[  168s] AssertionError: 0.0007986779999953342 not less than 0.00042071000000021286

(this is the complete build log on i586 with all packages used and steps taken to reproduce).

And this happens on aarch64:

[  164s] ======================================================================
[  164s] FAIL: test_instance_check_performance (tests.test_performance.PerformanceTest)
[  164s] ----------------------------------------------------------------------
[  164s] Traceback (most recent call last):
[  164s]   File "/home/abuild/rpmbuild/BUILD/nptyping-2.0.1/tests/test_performance.py", line 26, in test_instance_check_performance
[  164s]     self.assertLess(second_time_sec, 0.0002)
[  164s] AssertionError: 0.0002229950000014469 not less than 0.0002

(again, the complete build log).

Any idea, what's going on, please?

ramonhagenaars commented 2 years ago

Performance tests are a bit tricky, because they can be influenced by external processes that eat up your CPU. That would be my first guess as to why these are failing.

The first one (on i586) I find especially peculiar: second_time_sec should be lower due to the use of caching. Maybe on that architecture, the lru caching is not optimally implemented or something?

ggardet commented 2 years ago

This is on build farms, so the CPUs can be very busy. I guess we could increase the threshold?

ramonhagenaars commented 2 years ago

Yes I would say so.