search

tcunit / TcUnit

An unit testing framework for Beckhoff's TwinCAT 3
Other
258 stars 72 forks source link

Wrong duration time (sometimes) calculated #226

Closed sagatowski closed 1 month ago

sagatowski commented 9 months ago

Describe the bug Sometimes the duration/time is wrongly calculated. This happens completely randomly and I have not been able to see why or any patterns. See screenshot below.

To Reproduce Run tests over and over again. Eventually it will show up.

Screenshots image

Software versions Current master of TcUnit. XAE 3.1.4024.50. TcXaeShell.

Run environment Virtual machine.

@stefanbesler

sagatowski commented 9 months ago

image

stefanbesler commented 9 months ago

Sorry, I am a bit busy at the moment, I can look into this somewhen next week.

Since this is so sporadic and the duration is off by such a large value, I think it has to do with this here

https://github.com/tcunit/TcUnit/blob/4a48bdd61fc06e126195ad3a7d970774aee9c834/TcUnit/TcUnit/POUs/Functions/F_GetCpuCounterAs64bit.TcPOU#L14

This should most likely be

SHL(DWORD_TO_LWORD(CpuCounter.cpuCntHiDW), 32)
TimSalzmann commented 9 months ago

Can you give me a hint? I run TcUnit verifier 1.2.1.0 hand have 118 faild test is that correct? I want to simulate the random error. image

sagatowski commented 9 months ago

@TimSalzmann

The expected behaviour is documented here: https://github.com/tcunit/TcUnit/tree/master/TcUnit-Verifier

So to answer your question: https://github.com/tcunit/TcUnit/blob/4a48bdd61fc06e126195ad3a7d970774aee9c834/TcUnit-Verifier/TcUnit-Verifier_DotNet/TcUnit-Verifier/Program.cs#L21

stefanbesler commented 9 months ago

Could reproduce it and solved it in the PR

sagatowski commented 1 month ago

Re-opening issue as the test-duration is still not calculated correctly for TEST_ORDERED(). See more details here:

Keeping this open until this issue is fully resolved to remember to close this for TcUnit 1.3.1.

stefanbesler commented 1 month ago

Maybe just revert the feature. I am not using TcUnit with ordered tests and I don’t have an example to debug the issue, it’s kinda hard for me to fix this, also I feels like ages since I worked on this. For a long time I am even mocking everything that is time dependent out of my tests to have all of them finish in 1 cyclic

sagatowski commented 1 month ago

Solved in https://github.com/tcunit/TcUnit/pull/231.