CPU to 99% when running lots of tests

[Roald87]

Currently I have a project with some 200 unit tests. When I run the tests the cpu which runs them goes to 99% and sometimes doesn't recover. The only way out it to reboot the machine.

How do you people handle running a lot of unit tests? I currently am using #95, to get around it. But sometimes I need to add more tests to the delayed ones, which is very time consuming. Furthermore I quickly loose track of which tests run with a delay.

[sagatowski]

Hi @Roald87! I too have some quite big projects with hundreds of unit tests, but I've never had a problem, so I'm curious of how this issue arises. When you say that the CPU is up at 99%, how do you measure that? When you say it doesn't recover, what do you mean more specifically? Does the TwinCAT kernel crash, or is it just the ADS communication that gets slow? Do you still get results? Are you running the tests on a separate (isolated) core? Have you tried changing the cycle-time?

[Roald87]

When you say that the CPU is up at 99%, how do you measure that?

I'm looking at the CPU usage under SYSTEM > Real-Time in my TwinCAT project.

When you say it doesn't recover, what do you mean more specifically?

When I activate the project on my local machine with isolated cores or on a separate PLC also on isolated cores, the CPU usage jumps to 99% and stays there. On my local machine the only solution is to reboot. The PLC sometimes crashes and restarts on its own. At other times the plc doesn't respond anymore (not even when I try to ping it) and the only solution is a power cycle to get it back.

Does the TwinCAT kernel crash, or is it just the ADS communication that gets slow?

I think the TwinCAT kernel crashes. I do not see any ADS communication.

Do you still get results?

Nope.

Are you running the tests on a separate (isolated) core?

They run together with other tasks on an isolated core.

Have you tried changing the cycle-time?

Yes that seems to solve the issue, but now some tests fail because they depend on the cycle time. But that should not be that much work to fix.

Maybe the problem was that the unit tests somehow never finish within the cycle, because almost all start at the same time. During each subsequent cycle, the unit tests restart and still can't finish and therefore get stuck in an infinite loop?

[sagatowski]

@Roald87 Thanks for the response.

You're actually lifting an important point here, about cycle-times for tests. I guess unit tests shouldn't be too dependent on the cycle-time (and if so, it should be possible to configure the tests and/or the units under tests for this case), although I guess this is not always possible.

I would like to look into this issue, although I don't know how to recreate the problem. Is it possible for you to submit an example project that re-creates this?

I don't want to keep this ticket open for too long, but would like to find a solution, or at least add something to the FAQ on TcUnit.org to consider this.

[Roald87]

I can have a look if I can recreate the problem. You mean only the cycle time issue right? Not the 99% CPU one I guess?

[sagatowski]

@Roald87 That's right!

[Roald87]

Here is an example of a project which has a ramp which fails depending on the cycle time. With a base time of 1 ms and 10 cycle ticks is passes. However, if you for example set the cycle ticks to 5 or 20 it fails.

RampGenUnitTest.zip

[sagatowski]

Hi @Roald87 ! Sorry, I can't reproduce the error. If I change the parameter: tCtrlCycleTime:=T#5MS, tTaskCycleTime:=T#5MS To the cycle-time of the task, it is always succesful whether I run it at 5ms or 10ms.

[Roald87]

But if you don't change tCtrlCycleTime and the tTaskCycleTime to match the cycle time it will fail right?

This solution means that you would need to read out the cycle time and pass these parameters to the ST_CTRL_RAMP_GENERATOR_PARAMS before running the ramp. I couldn't find a solution how to get the cycle time from the task. Is there a way to do this?

[sagatowski]

@Roald87, yes there is! See: https://infosys.beckhoff.com/english.php?content=../content/1033/globaldatatypes/714821259.html&id=1449094368155551069

And example of how to use it: https://alltwincat.com/2017/11/05/first-cycle/

[Roald87]

Thanks it worked! I remember once looking for this, but I couldn't find anything. I guess this issue can be closed now?

[sagatowski]

Happy you got it to work! Yes, this issue can be closed now, as long as we solve the issue #95 that is related to this one.