Tests failing with too many methods

[Arkonos]

Build 4024.10 (can install another if you want me to) TcUnit 1.1.0.0

I have a test FB that contains 105 near identical methods, each containing 15 Asserts (7 AsserEquals_Real, 7 AssertEquals, 1 AssertEquals_BOOL).
When I run all 105 of them, I get errors that go away when I comment out any two of the methods from being called. Error: (I've shortened the time stamp and removed the always identical task/port column.)

MSG | :42 899 ms | Test name=SF123P23  
MSG | :42 919 ms | Test class name=PRG_TEST.fb_massFlowGUI_test  
MSG | :42 939 ms | Test status=PASS, number of asserts=15  
MSG | :42 959 ms | Test name=SF123P12À
    
MSG | :42 979 ms | Test class name=PRG_TEST.fb_massFlowGUI_test
MSG | :42 999 ms | Test status=FAIL, number of asserts=0
MSG | :43 019 ms | Test assert message=

MSG | :43 039 ms | Test assert type=UNDEFINED
MSG | :43 059 ms | Test name=SF123P123
MSG | :43 079 ms | Test class name=PRG_TEST.fb_massFlowGUI_test
MSG | :43 099 ms | Test status=FAIL, number of asserts=40105
MSG | :43 119 ms | Test assert message=Óÿÿ
MSG | :43 139 ms | Test assert type=UNDEFINED
ERR | :43 159 ms | ==========TESTS FINISHED RUNNING==========
ERR | :43 179 ms | Test suites: 1
ERR | :43 199 ms | Tests: 105
ERR | :43 219 ms | Successful tests: 103
ERR | :43 239 ms | Failed tests: 2
ERR | :43 259 ms | ======================================

The last few lines of the testing _FB_massFlowGUItest read

159 | SF123P12();
160 | SF123P13();
161 | SF12323();
164 | SF123P123();

So things start to get really wrong after line 161. The individual tests should be fine, after all they complete when any others are not performed. "Óÿÿ" can not be found in the entire solution, including external sources. The error also appears when I comment out one method and all but one of the asserts in a second. Adding a second test FB with 10 methods does not have an influence. All methods need to run for multiple scans, increasing the timer duration does not change this behaviour.

Here is the solution. It's under 3 MB and uses a PLC and HMI project. The latter is not needed for testing.

Thank you for your work.

[sagatowski]

Hi @Arkonos ! Thanks for your report. The example solution you provided does not compile. Can you look into it?

[Arkonos]

_PLC/Tests/FB_massFlowGUItest line 4 should read F1(); instead of Fs1(); Guess I didn't hit ctrl properly, apologies. I've reuploaded the fixed archive, but I think changing it manually is quicker.

[sagatowski]

I've re-downloaded the file, it still does not compile (changed Fs1 to F1). I also get an error:

[sagatowski]

Sorry, my mistake. I used an early pre-release of TcUnit. Using 1.1 now, will investigate the issue.

[sagatowski]

I've found the issue. The problem is that you have more than 100 tests in a single test suite. There is a configuration parameter in TcUnit to allocate memory for every test suite. Default is that the maximum is 100 tests for every test-suite, this can be changed however by changing a parameter for TcUnit. You made me realise it might be a good idea to add this to the FAQ, which I've now done: https://tcunit.org/frequently-asked-questions/?Display_FAQ=957

After changing the parameter to 110, the output looks like this:

I would also reconsider to splitting up your tests into different test suites if possible!

Good luck, and if you are happy with the answer you can close the issue :smiley:

ps. If you like the project, please star it, so that it's more likely others will find it too :wink: ds.

[Arkonos]

I thought I tested that when I ran 102 test successfully. Thank you very much for looking into it.