XogenyTest tests failing

[bilderbuchi]

Hi @mtiller!

I'm currently looking for a good (free/open source) package with which to write unit and integration tests for code I'm writing.

I have tried to run the tests for XogenyTest1.1 included in the current Openmodelica nightly, but many of them fail.

If XogenyTest is unmaintained or otherwise not suitable anymore, could you recommend another suitable testing framework? I have already found a couple but none of them seem promising/attractive.

Also, did the TestCase annotations ever really take off in Modelica? The only reference I found is in some compliance tests in OM, but those annotations look different...

Details

The test run details (I could only run those tests which are models. I don't know why some are functions, but there's no nice way I know of to run functions in OpenModelica, only models):

---

XogenyTest.Tests.Trajectory.CheckSuccess, .CheckFailure2 and .CheckFailure3 fail with something like this:

[2] 12:04:51 Translation Notification
It was not possible to analyze the given system symbolically, because the relevant equations are part of an algebraic loop. This is not supported yet.

[3] 12:04:51 Translation Warning
Assuming redundant initial conditions for the following 1 initial equations:
         $PRE.check_x.cur = 1

[4] 12:04:51 Scripting Notification
function BackendEquation.solveEquation failed w.r.t check_x.cur 
ALGORITHM: 0. algorithm 
  assert(check_x.expected[1,1] >= time, "Some trajectory points precede the simulation."); 
  check_x.cur := 1; 
  if check_x.expected[1,1] >= time and check_x.expected[1,1] <= time then 
    XogenyTest.assertValue(x, check_x.expected[check_x.cur,2], check_x.eps, ""); 
    check_x.cur := 2; 
  end if; 
  $whenCondition1 := check_x.cur <= 4 and time >= check_x.expected[check_x.cur,1]; 
  $whenCondition2 := terminal(); 
 (dynamic) 
=================== 

and

LOG_INIT | error | <p>The initialization problem is inconsistent due to the following equation: 0 != -1 = 1.0 - $PRE.check_x.cur</p>

I'm kinda assuming it's because of the double definition of cur since 0d3df1211ef74fcf313ee842f18c541230a52f4c

---

XogenyTest.Tests.BecomesTrueAt.CheckFailure4 crashes the process after reporting the failed assertion

stdout | error | <p>Process crashed<br>
Simulation process failed. Exited with code -1073741819.</p>

---

XogenyTest.Tests.BecomesTrueAt.CheckFailure2 and .CheckFailure3 reports an untreated assertion after correctly reporting that the tested assertion failed.

---

XogenyTest.Tests.Average.CheckSuccess, CheckFailure2 and .CheckFailure3 show variations of

assert | warning | <p>The following assertion has been violated at time 0.016000<br>
time &lt;= check_x.start</p>
assert | error | <p>The simulation started after the interval.</p>
stdout | warning | <p>Integrator attempt to handle a problem with a called assert.</p>
assert | warning | <p>The following assertion has been violated at time 0.008000<br>
time &lt;= check_x.start</p>
assert | error | <p>The simulation started after the interval.</p>
stdout | info | <p>model terminate | Simulation terminated by an assert at time: 0.008</p>
LOG_STATS | info | <p>### STATISTICS ###</p>
stdout | error | <p>Simulation process failed. Exited with code -1.</p>

---

XogenyTest.Tests.ValueAt.CheckFailure2 detects an untreated assertion.

---

In XogenyTest.Tests.ValueAt.CheckFailure3 the process crashed after correctly reporting the tested assertion as failed.

[xogeny]

Ideally, the OM guys would use XogenyTest is there regression testing. But I'm guessing they don't (although I think they still use the Modelica by Example models in their regression tests.

I really have no idea what the issue is here with OM. I haven't really used it in a long time and I certainly haven't been tracking the changes.

[bilderbuchi]

Ideally, the OM guys would use XogenyTest is there regression testing. But I'm guessing they don't (although I think they still use the Modelica by Example models in their regression tests.

Yeah, they test some of the libraries they ship, but XogenyTest is not part of them. @sjoelund do you know why OM does not test XogenyTest? (Interestingly enough, they run tests for @kdavies4' FCSys, although the tests have been failing for ~years, and the library does not actually support OM).

Michael, do you think this is an OM issue? Do the tests currently run correctly using e.g. Dymola? (I can't check :-/)

[xogeny]

I have no idea if it works with Dymola. I haven't used Dymola in years.

[bilderbuchi]

I see, what do you use currently?

[xogeny]

I have't actually done much Modelica work in a while. My main work now is building web-based engineering analysis tools. This means creating user interfaces, web APIs, backend services, interactions with databases, etc. It has been keeping my very busy lately.

[bilderbuchi]

Ah, that sounds interesting as well. So, are you interested in receiving bug reports/fixes/PRs for XogenyTest? I'm asking to calibrate expectations.

[xogeny]

I created several repos on GitHub in the hope that people would start building on them. I'd be happy to get bug reports, fixes, PRs, etc. I don't plan on doing any development work here (for now). I'm mainly trying to "shepherd" things along.