wjwwood / serial

Cross-platform, Serial Port library written in C++
http://wjwwood.github.com/serial/
MIT License
2.13k stars 1.04k forks source link

tests: Works on travis, fails on my buildbox. #232

Open leamas opened 4 years ago

leamas commented 4 years ago

This is really about the #231 PR. However, since the current master doesnt build, there is no way to reproduce test errors there.

On #231, commit 6dd5782, the run_tests completes successfully on travis. However, on my local test box the same test fails, with lots of messages like

timer_tests.short_intervals
/home/mk/cxx-serial/cxx-serial/tests/unit/unix_timer_tests.cc:24: Failure
The difference between r+1 and 0 is 11, which exceeds 1, where
r+1 evaluates to -11,
0 evaluates to 0, and
1 evaluates to 1.

My local box is pretty fast, a Ryzen 7 gen3. OTOH, the Debian machine I use for testing is a vm under VirtualBox.

But then again, I guess the travis build also runs on a vm.

leamas commented 4 years ago

After porting the tests to cppunit I can run the tests on my native, Fedora without any hypervisor layer. Same results, slightly better formatted:

Test Results:
Run:  2   Failures: 2   Errors: 0

1) test: SerialTimerTest::testOverlappingLongIntervals (F) line: 76 unix_timer_tests.cc
double equality assertion failed
- Expected: -14
- Actual  : -36
- Delta   : 5
- check 2 on 0

2) test: SerialTimerTest::testShortIntervals (F) line: 42 unix_timer_tests.cc
double equality assertion failed
- Expected: 0
- Actual  : -68
- Delta   : 1

It's a bit unfortunate that the delays are random and thus not possible to reproduce.

leamas commented 4 years ago

The tests are broken since they assume that for example usleep(5000) actually creates a 5 ms delay. However, there are no such guarantees, the delay is never shorter but typically somewhat longer.

The tests needs to be patched taking the actual, elapsed time into account rather than blindly trusting usleep()

EDIT: typos

leamas commented 4 years ago

Attaching updated tests, These are cppunit tests, not directly usable in current, catkin context. They also relies on C++11, which somehow makes the timer tests obsolete (since C++11 has platform-indepented timers built in). Nevertheless, enclosed tests works on both Linux and Macos, basically validating the pre-C++11 code with C++11 primitives.

unix_timer_tests.cc.gz