search

tcunit / TcUnit

An unit testing framework for Beckhoff's TwinCAT 3
Other
258 stars 72 forks source link

TcUnit returns erroneous results with 4020.32 #28

Closed sagatowski closed 4 years ago

sagatowski commented 5 years ago

When running the example project (simple) with 4020.32 the following result is returned:

==========TESTS FINISHED RUNNING==========
Test suites: 0
Tests: 0
Successful tests: 0
Failed tests : 0
======================================

Obviously this is not correct, and for some reason it seems the framework returns the wrong result with 4020.32 (or possibly even all of 4020)?

sagatowski commented 4 years ago

This was trickier to fix as the latest version of Win10 does not allow you to run 4020.32, but I rather need a PLC which doesn't have the latest Win10 patches.

sagatowski commented 4 years ago

Finally I've managed to get hold of a PLC with 4020.32 working (old enough version of Win10). It seems the framework is working now, except for the usecase when using the Assert_Equals(Any, Any). Then the assertions fail, even though the expected and actual values are the same: image

This screenshot was created using 4020.32 both on the developer machine (virtual machine with ONLY 4020.32 and 4020.32 on the PLC), and running the IO-Link/Advanced example project from TcUnit.

This needs to be investigated. Running the tests the final result is: Error 98 2020-01-07 08:10:45 672 ms | 'PlcTask' (350): | ==========TESTS FINISHED RUNNING==========
Error 99 2020-01-07 08:10:45 692 ms | 'PlcTask' (350): | Test suites: 5
Error 100 2020-01-07 08:10:45 712 ms | 'PlcTask' (350): | Tests: 17
Error 101 2020-01-07 08:10:45 732 ms | 'PlcTask' (350): | Successful tests: 9
Error 102 2020-01-07 08:10:45 752 ms | 'PlcTask' (350): | Failed tests: 8
Error 103 2020-01-07 08:10:45 772 ms | 'PlcTask' (350): | ======================================

sagatowski commented 4 years ago

Problem found. The problem lies in the fact that we are passing an ANY-reference twice! First to AssertEquals(ANY, ANY), and then inside the AssertEquals-method we instantiate the FB_AnyComparator, which takes the same ANY, ANY as inputs. TwinCAT doesn't seem to like this, and it might be the case that it can't handle it. 4022 handles it differently, although it too seems to handle the ANY.TypeClass wrongly, as if you for example declare two BYTEs, it sees them as ENUMS.

The safe way to go is to put the logic of FB_AnyComparator directly into FB_AssertEquals, as it's anyway the only place of where it's used

sagatowski commented 4 years ago

Solved in commit c50dcd8. Verified using a Beckhoff CX5140 running TwinCAT 4020.32, IDE running TwinCAT 4020.39. Also verified using the TcUnit-Verifier (TwinCAT/.NET) using XAE remote manager for 4020.39.