libnonius / nonius

A C++ micro-benchmarking framework
https://nonius.io
Creative Commons Zero v1.0 Universal
358 stars 47 forks source link

faster_than_clock.c++ test fail on Windows #74

Open bitshifter opened 7 years ago

bitshifter commented 7 years ago
test/faster_than_clock.c++:22: FAILED:
due to unexpected exception with message:
  could not measure benchmark, maybe it was optimized away

This test is failing for me when compiled with msvc2015 and mingw64 g++. I've tried it on multiple machines. It seems to be OS specific rather than compiler or hardware.

rmartinho commented 7 years ago

That is very strange, as the test uses a fake clock. Maybe this is not propagated properly across calls :|

rmartinho commented 7 years ago

Right, figured that out. However, that this failed on Windows specifically means that I can use it to reproduce another bug I've seen before. I'll have a usable Windows machine soon and I'll look into this.

bitshifter commented 7 years ago

I also reproduced it on Windows running in a VM on Linux if that's any help.

bitshifter commented 7 years ago

The test still seems to be failing on Windows with changes in ddddadf.

bitshifter commented 7 years ago

I've done some investigation into this. I think the problem is uninitialized variables on execution_plan. In particular the warmup_time member which is of type std::chrono::nanonseconds. If I print that out before the test runs it's different every time. It seems like there's no sensible default constructor being generated for execution_plan's duration values, if I add one like this:

execution_plan() : iterations_per_sample{}, estimated_duration{}, warmup_time{}, warmup_iterations{} {}

then the test starts passing.

I can understand the int values not being initialized, but the duration values look like they should have a default constructor - although I'm not sure what value the default should be initialized to.

bitshifter commented 7 years ago

OK, looking at the documentation on MSDN this is expected behaviour - the default constructed value is uninitialized - https://msdn.microsoft.com/en-us/library/hh874737.aspx?f=255&MSPPError=-2147217396

edit: it's strange that this is fine on Linux but not Windows, even when building with mingw64 g++ which has exactly the same chrono header as what I'm using on Linux. I can only assume it's some difference between how executables are linked or run on each platform.

bitshifter commented 7 years ago

Continuing my novella on this topic; I discovered that these uninitialised variables are reported when I run valgrind ./bin/test on Linux.