search

vipm-io / caraya

Assertion and unit test framework for LabVIEW
Other
59 stars 33 forks source link

Caraya and NI TestStand #96

Open kosist opened 4 years ago

kosist commented 4 years ago

Once I had the following use case. There was a need to execute unit tests from TestStand. And the idea was not to launch some script/CLI to execute unit tests, but that each step in TestStand will be unit test VI itself. As a proof of concept, I've tried to create unit test VI using Caraya. Each VI contained "Define Test.vi", and Assertion VIs. But, when code module was executed from TestStand (via Run -> Single Pass), Caraya UI was opened, but didn't contain any results. I've closed it manually, run sequence again - and no UI was opened, at all. After code modules were unloaded, (via File -> Unload Code Modules), behavior was the same - first time GUI without results, then - no GUI. In order to skip UI displaying, I've created additional VI which defined test suite, and actually called multiple unit test VIs. When it was called from TestStand, no UI was opened (which is correct), but on next test iteration the following error occured:

Caraya.lvlib:Test Suite.lvclass:Define Test Suite (Default).vi A test is already defined. Nested Test Suites are not supported LabVIEW: (Hex 0x2A) Generic error.

So somehow, calling Caraya VIs from TestStand causes that Caraya does not finish execution properly, or something like that - b/c either test results are not displayed, or it registers test suite twice, etc.

Of course, this is not proper usage of the toolkit, but anyway it is quite strange what could be the reason of such a behavior...

francois-normandin commented 4 years ago

@kosist I'm not surprised that it is not as straightforward as it could be, and I apologize for my lack of TestStand knowledge. Hopefully my questions and suggestions make some sense.

Up front, I see two symptoms from your description that point to a problem of application reference (the UI shows up blank, and Test Suite that stays opened).

For the blank test UI, my gut feeling is that TestStand executes your test VIs in a distinct application reference than the UI. So two sessions are opened, one where the tests are registered and run silently, and another one where no tests are run but the results are displayed. That should be fairly easy to confirm; if you output a test report, it should get created nonetheless. We'll come back to that part of the problem later.

The second symptom is the nested test suite error. I assume here that you are using the Define/Destroy Test Suite methods to wrap the defined tests. You are hinting that execution is not completing properly and I agree. When you create a Test Suite, it must be destroyed before another one can be created, otherwise we get a nested Test Suite and this is not supported.

So I'm wondering if the call to Destroy Test Suite gets enough time to complete it's asynchronous exchange with the Test Manager before TestStand kills the thread... That could hint at how to solve the problem in Caraya.

Here's my suggestion: Create two tests called sequentially from TestStand. Each test in LabVIEW would do the complete sequence described below:

If the nested test suite error is still there, then we know the framework has a bug in handling application references. If it disappeared, then we need to fix the part where the destruction of the test suite is interrupted by TestStand before it completes.

Alternatively, I'm trying to think of your use case and I believe there's a parallel to be made with use case #95 . I'll try an example code this weekend and see if it's pertinent to investigate this in the same context.