Unit test in separate project using TcRunner

[vegarosthus]

Hi all,

At my workplace, we want to keep our production code as clean as possible. This means we don't want the TcUnit library reference to be compiled down into our libraries (the specific reason for this is that it seems that some hardware (small PLCs such as the CX7000 etc.) don't have the sufficient memory to run our biggest libraries if TcUnit is loaded into it as well).

Therefore, I tried to separate out our unit tests into a separate test project in the same solution file as our library code. In this project i reference TcUnit and our library, and run the test on the referenced blocks. Now, I want to run this on a remote VM using TcRunner, but when I do this (through Azure Pipelines), I run into the obvious problem that the library I try to run tests on is not available. How can I can I compile and save my (updated) library on the VM, before referencing it with my test project? Is this even posssible, or am I chasing rainbows here?

[densogiaichned]

Hi @vegarosthus,

if possible, consider using Variant Manager. We use it for exactly this usecase. We have two variants, Unittest and production. In variant production, tcunit and all code belonging to unittest is not compiled.

[runtimevic]

Hi all,

At my workplace, we want to keep our production code as clean as possible. This means we don't want the TcUnit library reference to be compiled down into our libraries (the specific reason for this is that it seems that some hardware (small PLCs such as the CX7000 etc.) don't have the sufficient memory to run our biggest libraries if TcUnit is loaded into it as well).

Therefore, I tried to separate out our unit tests into a separate test project in the same solution file as our library code. In this project i reference TcUnit and our library, and run the test on the referenced blocks. Now, I want to run this on a remote VM using TcRunner, but when I do this (through Azure Pipelines), I run into the obvious problem that the library I try to run tests on is not available. How can I can I compile and save my (updated) library on the VM, before referencing it with my test project? Is this even posssible, or am I chasing rainbows here?

Hello, @vegarosthus I am interested in learning how to create pipelines and artifacts for TwinCAT with Azure, where can I find information or an example to start? Thanks for everything.

[vegarosthus]

Hi @densogiaichned,

Tanks for your reply! I didn't know about the Variant Manger, but it looks like the exact thing that we want. But is this supported by TcUnit-Runner if we want to automate this, i.e. selecting the test variant to run test on our remote VM when we check in our code?

Anyway, thanks for the tip. Looks very interesting.

@runtimevic Regarding the use of Azure Pipelines to facilitate the automation of unit tests, we used the example on the bottom of this page. We chose to create a cloud based VM as a testing machine, so we installed TwinCAT XAE and TcUnitRunner on it and added it as a self hosting agent to Azure Pipelines. This agent is specified by the .yaml file provided by you in your code repository. In our case, for reasons I can't recall, we had to replace the

pool: 'tcunit'    # self-hosted agent

in the example code with

pool: 
    name: AC4Build   
    demands:
    - agent.name -equals TcUnitTests # self-hosted agent

where AC4Build is our agent pool and TcUnitTests is the name of our VM/agent.

Now, assuming you've created some unit tests in your project using TcUnit, and assuming you've created a pipeline for your repository, the .yaml file will execute the test on your agent, and produce some test statistics.

Hope this helps you get going.

[kumaraswamygaviyappa]

Hello @vegarosthus , May this be useful for you. #93

Hello @runtimevic, Could you please provide some details about Variant Manager. when i googled, got may links , i am bit confused which one is correct. Thank you for your help.

[densogiaichned]

Hello @vegarosthus , please see PR #24 by @Hopperpop.

[vegarosthus]

@densogiaichned Ok! Cool, thanks!

[sagatowski]

I'll add a question to the FAQ regarding this. Thanks.

[sagatowski]

@kumaraswamygaviyappa Regarding variant management, this is what is discussed.

[sagatowski]

Actually, I won't add this to the FAQ. The discussion will continue in PR #24 by @Hopperpop. @vegarosthus My understanding is that PR #24 would fix what you need?

[vegarosthus]

@sagatowski In theory, yes. However, after trying this out for a couple of days now, when I run the program (built from the PR) on my unit tests I occasionally get this error message when parsing the test results

or this error message

or something similar.

It is hard to get a stable recreation of the bug, as I can run the program on the same unit tests back to back several times, and get these error messages, say, 3 or 4 out of 5 times, while the rest of the times it finishes clean. I don't know it it is something in my TwinCAT project that does it, or if it's something in the TcUnit-Runner code, but the tests run fine 10 out of 10 times when executed locally directly from TwinCAT.

Have anyone experienced these errors before?

[sagatowski]

@vegarosthus This seems to be a separate issue, not related to this issue, right? Did you have this issue prior to the code in this PR, or do you think it was introduced with this PR?

[vegarosthus]

@sagatowski I tried building the program from the 0.9.3.0 tag patch, and I haven't managed to get the error message after running the program 8-10 times in a row. That could perhaps indicate that the issue was introduced with the PR?

[densogiaichned]

@vegarosthus I don't think that the PR is responsible for those error messages. They look like the double items ErrorList message issue, i.e two TC projects in one solution.

Workaround for double items ErrorList:

• replace https://github.com/tcunit/TcUnit-Runner/blob/bbf93945452221303f97d34221128e13e8deb1e3/TcUnit-Runner/TcUnitResultCollector.cs#L124
• with Enumerable.Distinct

  foreach (var item in errors.Distinct().Where(e => e.ErrorLevel == vsBuildErrorLevel.vsBuildErrorLevelLow)) 

[vegarosthus]

@densogiaichned Thanks for your suggestion, but it didn't seem to help. It is correct that we have multiple projects in one solution, but for now, only one of them contains a task with unit tests that will be run by the program. I had hoped that I could add the same task to the other projects at and run unit test in them as well at later time. But for now only one project is run, to simplify matters.

Anyway, the same error messages show up after applying the workaround. Can the other projects interfere somewhow even though they contain no tasks?

[sagatowski]

@vegarosthus There is a similar problem here: https://github.com/tcunit/TcUnit/issues/92 There is an example project there referenced: https://github.com/DavidHopkinsFbr/TcUnitDuplicateMessageOutput

My understanding is though that everything works fine if you run the tests locally (without TcUnit-Runner) though, no?

[vegarosthus]

@sagatowski Yes, everything runs fine if I run the tests "locally" in TwinCAT. It seems like it's quite similar to this issue. I applied this PR and that actually did seem to do the trick (with some small modifications to the .bat file). I haven't had the time to test this extensively, though. I hope to get to do so sometime next week.

[Bulkje]

Oh haha I guess my .bat file was still for my installation sorry XD.

Regarding the compiling of a library, I think it's possible with the DTE. You could make a powershell script that compiles the library and then call the .bat with it to launch TcUnit-Runner.