search

tcunit / TcUnit

An unit testing framework for Beckhoff's TwinCAT 3
Other
273 stars 75 forks source link

ARMv7 compatibility and weird issues #148

Closed sagatowski closed 2 years ago

sagatowski commented 3 years ago

Hello, I've been trying to use TcUnit in my CX8180 project. Unfortunately, I've run into some weird issues and started to wonder if TcUnit is compatible with ARMv7 targets at all. I'm a twincat newbie so I'm not really sure whether the problem is in my libraries, settings or elsewhere and I'm at a loss as to why referencing a library which uses TcUnit causes all kinds of issues - unit testing code shouldn't even execute in such case, right?

At first, I developed and tested 2 libraries on local x64 twincat runtime and everything seemed ok. The moment I reference my libraries in CX8180 project (even if nothing uses them!) I lose ability to set breakpoints, value forcing is unreliable and PLC seems to randomly skip instructions.

To troubleshoot the issue I've tried the SequentialTestSuitesAndTests example project which when compiled and activated on CX8180 causes the same issue - even though the tests execute properly it is impossible to set a breakpoint anywhere and flow control doesn't work.

What am I doing wrong?

Originally posted by @chiveicrook2 in https://github.com/tcunit/TcUnit/discussions/147

sagatowski commented 3 years ago

Waiting for ArmV7 PLC before being able to reproduce.

sagatowski commented 3 years ago

Moved this to release 1.2. Was planning to release 1.2 next week but I think it's important to straighten this one out, as more and more people seem to be using the ARM-based PLCs (and probably even more once CX7000 is released).

sagatowski commented 3 years ago

Moving this to 1.3.0.0 because of time constraints. If anyone else has time to look into the issue, it would be highly appreciated.

FirmacChris commented 3 years ago

Just to give some feedback, I currently have what appears to be a very similar issue. TwinCAT V3.1.4024.20 (I'm using the XAE Shell, I never tried the issue in a VS install). TcUnit 1.2.0.0

The controller is a CX9020 running TwinCAT CE7 (ARM-V7).

Debugging/build/breakpoints worked fine using the XAR enviroment on my laptop (windows 10) but when I try to put it into my PLC I wasn't able to set breakpoints. Everything else seemed to work.

Trying to set a breakpoint anywhere in the program just gave a generic "PLC Control breakpoint could not be set on target" error message.

Removing TcUnit, and all references in the program, and then restarting XAE allowed me to set breakpoints again.

oswin02 commented 3 years ago

Could not reproduce any of these problems on my setup. Can anyone provide me a project where these issues appear? There is a problem with the remote manager in TC 4024.20, so I will not update to this version.

My Config:

Engineering PC: Windows 10 XAE Shell 3.1.4024.17 (tested also with remote manager to 4024.7)

Test PLC: CX9020 TC 3.1.4024.12

TcUnit 1.2.0.0

chiveicrook2 commented 3 years ago

On my CX8180 the issue is present in even the most basic projects, for example in all projects from tcunit example repos (https://github.com/tcunit/ExampleProjects). So far tested with 4024.7 and 4024.12 (both on engineering PC and CX) and TcUnit 1.1. Unfortunately, lack of time stopped me from debugging this further and I simply remove unit-tests before deploying to PLC...

oswin02 commented 3 years ago

I could reproduce the issue on my plc with the SimpleTcUnitExampleProject and found a workaround.

Adding 'UnitTest' as condition for conditional referencing of the TcUnit library and adding 'UnitTest' in the compiler defines in the compile settings solved the brakepoint issue for me.

https://infosys.beckhoff.com/content/1033/tc3_plc_intro/2533017227.html?id=8439395065138194985 https://infosys.beckhoff.com/content/1033/tc3_plc_intro/4215784459.html?id=1823229847829719237

sagatowski commented 3 years ago

@FirmacChris and @chiveicrook2 Can you confirm that this work-around is OK/working? I don't have an ARM-compatible controller on hand so I can't verify this. Would it be enough to add this to the FAQ of TcUnit?

Barteling commented 2 years ago

Working on a CX9020 today and I had the same issues as discussed above. The work around mentioned by @oswin02 did not help.

Some thoughts and remarks:

candacespencer commented 2 years ago

I am trying to test on on a CX7000 (ARMT2) and any project that references TcUnit will cause the project to build but fail to load on the controller.

I get the following error when trying to activate the project: 'Port_851' (851): PLC: Error (hr=0x98410027) loading boot project.

sagatowski commented 2 years ago

Is it enough for the project to reference TcUnit for it to fail? (so not executing TcUnit)? Can you check the AMS memory allocation of the device with and without TcUnit?

Also, can you test to play around with the parameters related to memory allocation: https://tcunit.org/frequently-asked-questions/?Display_FAQ=957

candacespencer commented 2 years ago

I was able to get the application to load and run a few unit tests by adjusting the parameters. Thanks.

sagatowski commented 2 years ago

@candacespencer Great! Could you post a screenshot of the values you used to get it to run?

sagatowski commented 2 years ago

@Barteling Can you confirm if changing the parameters will fix the issue for your CX9020 as well?

maxxie85 commented 2 years ago

I modified the parameters and got it working on a CX8190 with the following settings afbeelding

For clarification the following parameters are reduced in size

Increasing any of these settings to the original value will bring the problem back.

sagatowski commented 2 years ago

Thanks @maxxie85! It seems this is simply a memory problem as per @Barteling suggestion, probably due to the limited memory in the simple WinCE-CXs compared to the bigger W7/W10 cousins. I will add the above information to the FAQ of the TcUnit website and close this issue afterwards.

sagatowski commented 2 years ago

I have updated the FAQ with an entry for this in case anyone else encounters the same problem.

ZigaJavornik commented 3 months ago

I know this is a closed issue, but how does one work with 3rd party library that uses TcUnit? The parameters are not changeable in that case, I had to exclude the library for now: image

sagatowski commented 3 months ago

I know this is a closed issue, but how does one work with 3rd party library that uses TcUnit? The parameters are not changeable in that case, I had to exclude the library for now: image

How do you mean? If you're using a library that is using TcUnit, why would you want to change the parameters of TcUnit?

ZigaJavornik commented 3 months ago

I know this is a closed issue, but how does one work with 3rd party library that uses TcUnit? The parameters are not changeable in that case, I had to exclude the library for now: image

How do you mean? If you're using a library that is using TcUnit, why would you want to change the parameters of TcUnit?

Well, If I attempt to download to a PLC that will be getting the upper errors, I would need to be able to either lower the test parameters or remove the reference due to lack of memory, is that not correct? From what I read, just a reference in the library manager is enough?

I was getting the mentioned breakpoints errors, I could not set it at all, using my own libraries, they also used Roalds TcError, which I also assumed is the problem. In the midst of debugging, and finding this in the FAQ, I had to just disable the tests from build and remove references from all the projects for now.

I will have to inspect further next week, this week I am on a different machine. But I have to note, once I change the parameters in my library that uses TcUnit, on my project that is using the created library, if I open the library manager, parameters are default values as well, despite me changing the value to a lower one inside the library project. Am I doing something wrong perhaps? I build, cleaned, installed, etc... No matter what, when i look in the library repo, the values are still at max, in the end I ended up getting "not enough memory" when I attempted to download to the PLC (CX8110) and I had to just remove the project reference as a whole, for now:

image

I-Campbell commented 3 months ago

to change library parameters, the tcUnit library needs to be added to the root level of the library manager in the final application. A library cannot influence the library parameters of another library. A bit annoying really, but workable.

ZigaJavornik commented 3 months ago

to change library parameters, the tcUnit library needs to be added to the root level of the library manager in the final application. A library cannot influence the library parameters of another library. A bit annoying really, but workable.

How do you go about using libraries and TcUnit then? I am not sure I understand how to deal with this.

sagatowski commented 3 months ago

@ZigaJavornik You only need to care about changing the parameters of YOUR library (that executes the TcUnit tests). If your library depends on another library that uses TcUnit tests, that library-project will have its own set of parameters that will be stored only for that library. This is however not something you need to care about because that librarys tests will not be executed in your project. Therefore you don't need to be able to change the parameters of another library (Which as @I-Campbell has correctly noted, is not possible).

oswin02 commented 3 months ago

I suggest deleting all tests before releasing a lib. Unit tests are not needed in a productive system.

ZigaJavornik commented 3 months ago

I suggest deleting all tests before releasing a lib. Unit tests are not needed in a productive system.

The whole point of tests is to have them run before a release is done. Deleting them, or disabling them, seems very unproductive way of going about it.

@sagatowski Even if I use my own library, as a released library, having TcUnit being referenced there, that is already the problem, as far as I can see. Do people just remove TcUnit when they release a public version to avoid this? I should add that I do not with to run unit tests on an ARM device, they will be ran on a virtual machine, I do however want to use these released libraries in production, of course.

sagatowski commented 3 months ago

I would NOT recommend to delete all tests prior to the release. The tests are excellent documentation of the behavior of your code that is released together with the library. I always ship the tests together with my POUs (albeit in a separate folder, like /test or alike in parallel to /POUs).

I don't understand why it is a problem to have TcUnit referenced in your library. Your libraries tests will not be executed in the context of whoever is using that library (whether that is another library or a PLC-executable). The parameterization of TcUnit will ONLY affect whatever tests are running in that library. It will NOT affect any other tests that might be in any other libraries as these will simply not be executed.

If it's simply a matter of that you don't want to show the world that you are using TcUnit, you can hide your reference if you want to.

ZigaJavornik commented 3 months ago

I would NOT recommend to delete all tests prior to the release. The tests are excellent documentation of the behavior of your code that is released together with the library. I always ship the tests together with my POUs (albeit in a separate folder, like /test or alike in parallel to /POUs).

I don't understand why it is a problem to have TcUnit referenced in your library. Your libraries tests will not be executed in the context of whoever is using that library (whether that is another library or a PLC-executable). The parameterization of TcUnit will ONLY affect whatever tests are running in that library. It will NOT affect any other tests that might be in any other libraries as these will simply not be executed.

If it's simply a matter of that you don't want to show the world that you are using TcUnit, you can hide your reference if you want to.

Perhaps I have moved away from what the problem was, sorry.

My only "problem" (if you could call it that) with TcUnit was that the PLC that referenced the libraries using it, could not work normally (breakpoints error and during download to the PLC there would be "not enough memory" message), probably due to the memory problems mentioned in this thread. My whole point of how to change parameters/removing TcUnit was my thinking, that if there is only a reference to the library, the memory issue already exists, since I was able to download and debug normally after i removed TcUnit reference from my libraries and re-deployed them and added new reference in the PLC program to these new library releases that no longer had TcUnit in the.

I would never want to hide the fact that there are unit tests, or that I use TcUnit, on the contrary, it is a bonus and I want to know that the code is tested and written with best practices in mind. And just like you mentioned one time, the best way to get to know functionality of a library is trough tests, as opposed to comments (which are also nice, but together, its a win win).

I will put a pause on this until next week, when I will have time to re-check this on the PLC it-self as I am on other projects for the entire week. I hope that I explained better now what my only problem was/is.

ZigaJavornik commented 2 months ago

I can confirm that as soon as my project references my own library, which uses TcUnit, I am unable to set breakpoints. I am using CX8110, and I add my own library to the project, that library also contains unit tests and includes TcUnit. @sagatowski what would your suggestion be? For now, I disable TESTS folder and remove TcUnit reference before doing a release.

sagatowski commented 2 months ago

@ZigaJavornik Did you try any of the suggestions in this thread? Also check this FAQ-entry: https://tcunit.org/#/faq?id=_12-i-have-problems-running-tcunit-on-a-armv7-controller-why

ZigaJavornik commented 2 months ago

I have, but as mentioned, when I import a library (my own) that uses TcUnit, it has default parameters, no matter if I change the parameters in the original project that uses TcUnit, you can see that here: image

I have read trough the post, but it does not seem to solve the issue. So, step by step:

If I understand correctly people ran tests on their ARMv7 devices, hence they were able to change the parameters to be able to run them without issues? I don't want to do that, all i need to do is use my tested libraries, which by default have TcUnit reference in it, and as mentioned, this automatically causes the bugs ...

sagatowski commented 2 months ago

@ZigaJavornik Thanks for clarification. I don't know why you can't change the parameters for a library that you're not referencing directly. In one way I think it makes sense as your application does not use that library directly (it has no references to it), so it should not affect your execution.

But your PLC-program is not running any tests, right? It's only your library that is running tests. So why do you need to change the TcUnit-parameters from the PLC-program? I don't think that will make any difference with the breakpoints.

Did you try to hide TcUnit completely? So it's not even visible in the PLC-program.

https://tcunit.org/#/faq?id=_4-is-there-a-way-to-hide-tcunit-in-my-libraries

ZigaJavornik commented 2 months ago

Yes, that is correct. I do not run any tests on the PLC, as it is a machine. I just have the library with tests included in the release. I will try by hiding the reference and see if it solves the problem. Do you perhaps have any clue if this is a "me" problem, or have you had similar experience before perhaps? I will let you know if I find a solution.

ZigaJavornik commented 2 months ago

I went ahead and read a bit about properties for libraries and I think this one might be what I am looking for :

image

If I understand correctly, I will put in some condition that will be false when instantiating it in the project and it should be disabled in that case - my assumption is going this way.

As for hide reference, it seems it only hides the reference from UI, it even says that compiler errors won't be shown and project might not compile if the error exists in the library, which leads me to believe that the library will still "exist": image

sagatowski commented 2 months ago

That's what @oswin02 was referring to and it solved his problem. Try it out.

Also try hiding the library.

ZigaJavornik commented 2 months ago

Well, I missed that comment. Perhaps adding it to the FAQ or even the readme would be really neat :) I will do both next week, thank you @sagatowski

sagatowski commented 2 months ago

Well, I missed that comment. Perhaps adding it to the FAQ or even the readme would be really neat :) I will do both next week, thank you @sagatowski

You're more than welcome to make a PR and make an addition to the FAQ for this. If this is updated the website is automatically updated.

ZigaJavornik commented 2 months ago

Once I confirm it fixes the the issue I will do that, cheers.

ZigaJavornik commented 2 months ago

@sagatowski An update, today I attempted to load with condition, but I still had problems. I will do hide reference and condition tomorrow, hope that is the solution - it could be the case since even disabling TcUnit in the library project, when imported as a library to a third project and looking at the references, TcUnit was enabled (the reference in the imported library) and looking as if it were a normal reference in the release. Sorry for the weird phrasing, I am not really sure how to write this 🙉

Otherwise I will probably go ahead and make a script to create a release by deleting TcUnit reference, generating the library and adding it once again.

ZigaJavornik commented 5 days ago

@sagatowski I have finally got around to being on Beckhoff again :) I think I can confirm that under no circumstances can you remove the TcUnit from being included - I used compiler defines and made it disabled that way as well as hide reference. This still resulted in the GVL data to be generated, I can confirm this from target browser and not a single library in my project displays TcUnit (this is most likely how globals work with Beckhoff?). I belive the only way to really not have any data from TcUnit is to delete it before release and only add it when producing the libraries. If i delete TcUnit in my library before I create the release, the GVL data from TcUnit is deleted when i download again.

image