Automatically disable all I/Os prior to running tests

[sagatowski]

A TwinCAT project can be either a full-blown executable (that you can do "activate configuration" with) or a library. Usually unit tests are added to the library, but sometimes it might be benefical to add the tests directly to the main program and adding a separate task and separate program for the tests and run that instead. This is today possible with the TcUnit-Runner. What might cause a problem however are the different I/Os (such as an EtherCAT master) which won't work if the hardware is not there.

This can be solved by TcUnit-Runner automatically disabling all I/Os prior to executing the unit tests. This can be achieved through the TwinCAT automation interface.

[Beidendorfer]

I think we have to search and deactivate all EtherCat Masters. There may be programs with more than one Ethercat Master. When the Ethercat master is deactivated, the underlying terminals are also deactivated

The question is whether we want to reactivate the Ethercat Master after the build?

Then we have to save the current configuration before deactivating it and restore it at the end

[Hopperpop]

Why not use the variant manager in TwinCat for this? You make different configuration for the environment you are running on, and use attributes to enable/disable code. This way you can easily switch between running the real project on the hardware, or running a simulation/unit test. Some people would maybe want to run the unit test on real hardware but with different configuration than their production hardware. Disabling all I/O won't work for them, but with variants they could make a specific configuration for the unit test.

(I was basically searching if TcUnit-Runner had a parameter for variants, but I ended up here.)

EDIT: It can also be used to include/exclude the TcUnit library from the project.

[sagatowski]

If you're starting to involve real real hardware into the unit tests, I'm not sure I want to call them unit tests anymore. Unit tests should be easy to write, and more importantly, easy to run. If you start to add external dependencies (be it a database, filewriter, I/O) I think we're outside of unit-test-land but integration-test-land. I've done unit tests where I've had dependencies to I/O, but the I have created an interface to the dependency and then mocked the I/O. I've written a tiny article (never finished part 2) about this: https://alltwincat.com/2018/05/23/mocking-objects-in-twincat/

This great book covers this topic much more greatly and with many more examples: http://www.growing-object-oriented-software.com/

Also, the variant manager is only available from a quite late version of 4022 (don't know which one though, maybe you do?) TcUnit (1.X) is compatible with 4020.0 and onwards. If we added support for the variant manager, we would need to check that the project is at least opened with version 4022.XY.

This issue is really not a big deal as long as you write libraries (which from the uses I've seen of TcUnit, this has mostly been the case).

[Hopperpop]

It's true that unit tests are better of without hardware, and wasn't the best argument. I'm just trying to imagine when we would build the next machine with a lot of motion/pneumatic, how we could integrate automatic tests to decrease debugging on the machine. For the pneumatics this would indeed be a mocking function block for simulations of the cylinders. But you still need to tell the program what to use: the mocking fb or the real implementation. With variants you could easily use conditional pragmas to switch between them. For the motion you need to switch over to simulated axis. It can be done with the automation interface or again with the variant manager. When you have all the hardware mocked, the machine could be simulated and tested. Testing if it correctly runs and doesn't cause any collisions.

It all boils down to the choice if you use the variant manager to switch between configurations or the automation interface. The variant manager looks simpler to implement with the most possibilities, but is indeed not supported in older version. (The Beckhoff website says it's supported from build 4024.)

[sagatowski]

@Hopperpop The possibility to select the real or mocked version of an object depending on which variant is used is a good point, I never thought of that. When I've been mocking objects I've declared the usage of the mocks in the tests (which anyway reside in a separate library, not the actual executable itself), but I guess some users of TcUnit might want to include the tests in the executable project directly.

Do you know how to access the variant manager from the automation interface?

One way to handle the limitation that it's only available from 4024 is to be explicit and inform (in documentation + software itself) that the "Variant-Option" is only possible to use from 4024.(0?) and later. In that case we would though have to make sure that:

• if anything < 4024.0 is loaded AND the option is provided, the user gets an error message
• if a variant is provided, and that variant is not available in the configuration, the user gets an error message

[Hopperpop]

I haven't worked with the automation interface yet. In the documentation I don't find any reference to it. Maybe it's not possible yet. But I will look further if I can find something.

[densogiaichned]

The current version of TCatSysManagerLib (V3.3.0.0) does support project variants, see ITcSysManager15 or ITcSysManager14.

[Beidendorfer]

Hi, here is the Link to the Automation Interface and "TwinCAT variant management" https://infosys.beckhoff.de/english.php?content=../content/1033/tc3_automationinterface/8204600715.html&id= I'm just thinking if the variant management doesn't belong more in a kind of implementation test instead of unit test. maybe TcImplementationTest???

But back to the actual questions: should a Possible Hardware in TcRunner be disabled to enable the test?

I see the following possibilities:

1.) If hardware is found, the test is aborted with an error.

2.) If hardware is found, it will be disabled. Info output in the log. No enable at the End of the Test

3.) If hardware is found, it will be disabled. Info output in the log and activated again after the test. Here must be looked which hardware was already deactivated before the Test. It gets complex if only single Cards are disabled.

I am for solution 2. it focuses TcRunner on the unit test. It would have the same behavior as the TC version. Here also the last version is taken (if not with options one is forced) and at the end the .tsproj with this version is stored.

[Beidendorfer]

furthermore I would like to test what happens when TcRunner wants to test a program that has 2 variants.

I could imagine that there is an error. Since the compiler does not know which version to build

[sagatowski]

I've created a separate issue for variant handling. Please continue discussion about variants in #25 or PR #24.

[sagatowski]

Continuing discussion from #24 and #25, that discussed disabling of I/O. I think it's better to discuss it here so we don't mix issues.

@Beidendorfer

I just want to tell you one more experience with the Variant Manager. I use many PLC programs with variants and Git. And I remember when I change the branch or repo and want to activate the configuration, there is a TcSystem Error 1000. This is related to the fact that no variant is selected. After selecting the variant everything runs without error. This indicates to me that the variant config is outside the project.

@Beidendorfer have you managed to confirm this? Is the last selected variant indeed saved outside of the project? I've used variants a little bit and I remember that the variant selection is sprayed all over various files in the project (especially if you have separate XTI-files for all I/O etc). In any case, if there is more than one variant in the project, the more important there is that you can actually select it using the suggestion that @Hopperpop has given if you want to run it with TcUnit-Runner. Indeed one way to solve it could indeed be to always create a variant that is without I/Os (all I/O disabled).

From what I've seen, most people that use TcUnit use it for testing separate libraries (that are not executables), and then you don't have the problem of I/Os at all.

Would still be nice to have this flexibility though.

Question is, if we decide to always disable I/O, should we handle it differently if we provide a variant? In other words, if a variant is provided, should that "override" the automatic disabling of the I/Os?

[densogiaichned]

Despite of having the variant feature, IMHO it would be nice to have, lets say another parameter to disable all I/Os 'manually', like you suggested:

This can be solved by TcUnit-Runner automatically disabling all I/Os prior to executing the unit tests. This can be achieved through the TwinCAT automation interface.

The feature is fairly new, well at least i didn't know about until this thread... And i personally find it a little bit tedious to enable and disable certain I/Os only for a unit test run. I would very much like to have an parameter, to disable all I/Os. at once, i.e. -x|--disable.

BUT
What if someone is using the variant manager to enable or disable also some parts of the code (compiler defines)? How to deal with that? The only feasible option would be to trigger the variant manager. And in that case should only the current activated variant be tested, or all variants one by one?

[sagatowski]

Despite of having the variant feature, IMHO it would be nice to have, lets say another parameter to disable all I/Os 'manually', like you suggested:

This can be solved by TcUnit-Runner automatically disabling all I/Os prior to executing the unit tests. This can be achieved through the TwinCAT automation interface.

The feature is fairly new, well at least i didn't know about until this thread... And i personally find it a little bit tedious to enable and disable certain I/Os only for a unit test run. I would very much like to have an parameter, to disable all I/Os. at once, i.e. -x|--disable.

BUT What if someone is using the variant manager to enable or disable also some parts of the code (compiler defines)? How to deal with that? The only feasible option would be to trigger the variant manager. And in that case should only the current activated variant be tested, or all variants one by one?

I would even stretch it and say that disabling all I/O should be the NORMAL case, in other words the -x flag needs to be provided if you DON'T want I/O. The I/O configuration is anyways outside of the realm of unit tests, but you should still be given the opportunity to include them if you want (althought it should not be the normal case).

If someone uses variants to disable some parts of the code that's fine. It's up to the developer to make sure that the right variant is set for the unit tests, whatever that might be.

[HansHagmanYaraMarine]

PLC is not my first language, excuse any grammatical errors :P

I know it can be a bit bothersome or not the TwinCAT way. but can't one do a search and replace for the network card settings?

[HansHagmanYaraMarine]

A work around with PowerShell to disable the card. Be mind full of the path below, but otherwise it should work. I use it in a Jenkins setting.

Write-Host ' ************************ Find Network card settings and disable it locally for test ************************'
$FilePath = Get-ChildItem ".\_Config\IO\*.xti"
Write-Host $FilePath
[XML]$xml = gc $FilePath
$xml.TcSmItem.Device.SetAttribute("Disabled", "true")
$xml.Save($FilePath)

[sagatowski]

This project is archived. Replacement for it is in the works.