added tellurium testing column to markdown table

[robertvi]

Now a new column in the markdown results file indicates whether the sedml file ran okay in tellurium or not. No output is tested, just whether tellurium ran okay or threw an error. A new option on the python script can limit how many of the test cases are processed, to facilitate debugging, as the entire suite now take an hour or so to run on my laptop.

[pgleeson]

Great. Any indication of what common features the tests that fail have?

[robertvi]

I'll maybe try adding an option to capture the test number and the exception into a log file, so we can analyse them

[robertvi]

Most errors are Unable to support algebraic rules.

>>> ENGINE ERROR for: tellurium with 00039/00039-sbml-l3v2-sedml.xml
Unable to support algebraic rules.  The formula '0 = -1 * k1 + S1 + S2' is not supported., at rrllvm::LLVMModelDataSymbols::LLVMModelDataSymbols(const libsbml::Model*, unsigned int)
--
>>> ENGINE ERROR for: tellurium with 00040/00040-sbml-l3v2-sedml.xml
Unable to support algebraic rules.  The formula '0 = -1 * k2 + S1 + S2' is not supported., at rrllvm::LLVMModelDataSymbols::LLVMModelDataSymbols(const libsbml::Model*, unsigned int)
--
>>> ENGINE ERROR for: tellurium with 00182/00182-sbml-l3v2-sedml.xml
Unable to support algebraic rules.  The formula '0 = -1 * k1 + S1 + S2' is not supported., at rrllvm::LLVMModelDataSymbols::LLVMModelDataSymbols(const libsbml::Model*, unsigned int)
--
>>> ENGINE ERROR for: tellurium with 00184/00184-sbml-l3v2-sedml.xml
Unable to support algebraic rules.  The formula '0 = add(S1, add(S2, -1 * k1))' is not supported., at rrllvm::LLVMModelDataSymbols::LLVMModelDataSymbols(const libsbml::Model*, unsigned int)
--
vagrant@ubuntu-jammy:~/repos/forks/SBMLShowcase/test_suite$ cat ./output.log | grep '>>> ENGINE ERROR' -A 1
>>> ENGINE ERROR for: tellurium with 00039/00039-sbml-l3v2-sedml.xml
Unable to support algebraic rules.  The formula '0 = -1 * k1 + S1 + S2' is not supported., at rrllvm::LLVMModelDataSymbols::LLVMModelDataSymbols(const libsbml::Model*, unsigned int)
--
>>> ENGINE ERROR for: tellurium with 00040/00040-sbml-l3v2-sedml.xml
Unable to support algebraic rules.  The formula '0 = -1 * k2 + S1 + S2' is not supported., at rrllvm::LLVMModelDataSymbols::LLVMModelDataSymbols(const libsbml::Model*, unsigned int)
--
>>> ENGINE ERROR for: tellurium with 00182/00182-sbml-l3v2-sedml.xml
Unable to support algebraic rules.  The formula '0 = -1 * k1 + S1 + S2' is not supported., at rrllvm::LLVMModelDataSymbols::LLVMModelDataSymbols(const libsbml::Model*, unsigned int)
--
>>> ENGINE ERROR for: tellurium with 00184/00184-sbml-l3v2-sedml.xml
Unable to support algebraic rules.  The formula '0 = add(S1, add(S2, -1 * k1))' is not supported., at rrllvm::LLVMModelDataSymbols::LLVMModelDataSymbols(const libsbml::Model*, unsigned int)
--

[robertvi]

(.venv) vagrant@ubuntu-jammy:~/repos/forks/SBMLShowcase/test_suite$ cat ./output.log | grep -A 1 'ENGINE ERROR' | grep -v -e '^>>>' -e '--' | grep -v -e 'Unable to support algebraic rules' | sort -u
'float' object is not callable
Mutable stochiometry for species which appear multiple times in a single reaction is not currently supported, species reference id: A1_sr, at virtual llvm::Value* rrllvm::ModelDataStoreSymbolResolver::storeSymbolValue(const string&, llvm::Value*)
Mutable stochiometry for species which appear multiple times in a single reaction is not currently supported, species reference id: A_sr1, at virtual llvm::Value* rrllvm::ModelDataStoreSymbolResolver::storeSymbolValue(const string&, llvm::Value*)
Mutable stochiometry for species which appear multiple times in a single reaction is not currently supported, species reference id: Q, at virtual llvm::Value* rrllvm::ModelDataLoadSymbolResolver::loadSymbolValue(const string&, const llvm::ArrayRef<llvm::Value*>&)
S4_stoich is not a named SpeciesReference, at const rrllvm::LLVMModelDataSymbols::SpeciesReferenceInfo& rrllvm::LLVMModelDataSymbols::getNamedSpeciesReferenceInfo(const string&) const
Unable to support delay differential equations.  The function 'delay(P1, 1)' is not supported., at llvm::Value* rrllvm::ASTNodeCodeGen::delayExprCodeGen(const libsbml::ASTNode*)
Unable to support delay differential equations.  The function 'delay(P1, k)' is not supported., at llvm::Value* rrllvm::ASTNodeCodeGen::delayExprCodeGen(const libsbml::ASTNode*)
Unable to support delay differential equations.  The function 'delay(Q, 1)' is not supported., at llvm::Value* rrllvm::ASTNodeCodeGen::delayExprCodeGen(const libsbml::ASTNode*)
Unable to support delay differential equations.  The function 'delay(S1, 0.1)' is not supported., at llvm::Value* rrllvm::ASTNodeCodeGen::delayExprCodeGen(const libsbml::ASTNode*)
Unable to support delay differential equations.  The function 'delay(S1, 1)' is not supported., at llvm::Value* rrllvm::ASTNodeCodeGen::delayExprCodeGen(const libsbml::ASTNode*)
Unable to support delay differential equations.  The function 'delay(S1, S1_stoich)' is not supported., at llvm::Value* rrllvm::ASTNodeCodeGen::delayExprCodeGen(const libsbml::ASTNode*)
Unable to support delay differential equations.  The function 'delay(S1, addtwo(0.5, 0.5))' is not supported., at llvm::Value* rrllvm::ASTNodeCodeGen::delayExprCodeGen(const libsbml::ASTNode*)
Unable to support delay differential equations.  The function 'delay(S1_stoich, 1)' is not supported., at llvm::Value* rrllvm::ASTNodeCodeGen::delayExprCodeGen(const libsbml::ASTNode*)
Unable to support delay differential equations.  The function 'delay(addtwo(S1, S2), 1)' is not supported., at llvm::Value* rrllvm::ASTNodeCodeGen::delayExprCodeGen(const libsbml::ASTNode*)
Unable to support delay differential equations.  The function 'delay(p2, 1)' is not supported., at llvm::Value* rrllvm::ASTNodeCodeGen::delayExprCodeGen(const libsbml::ASTNode*)
Unable to support delay differential equations.  The function 'delay(reset, 0.005)' is not supported., at llvm::Value* rrllvm::ASTNodeCodeGen::delayExprCodeGen(const libsbml::ASTNode*)
Unable to support delay differential equations.  The function 'delay(t1, timeconv * 3)' is not supported., at llvm::Value* rrllvm::ASTNodeCodeGen::delayExprCodeGen(const libsbml::ASTNode*)
Unable to support delay differential equations.  The function 'delay(t5, timeconv * 0.2)' is not supported., at llvm::Value* rrllvm::ASTNodeCodeGen::delayExprCodeGen(const libsbml::ASTNode*)
Unable to support delay differential equations.  The function 'delay(x, 0.2)' is not supported., at llvm::Value* rrllvm::ASTNodeCodeGen::delayExprCodeGen(const libsbml::ASTNode*)
Unable to support delay differential equations.  The function 'delay(x, 0.5)' is not supported., at llvm::Value* rrllvm::ASTNodeCodeGen::delayExprCodeGen(const libsbml::ASTNode*)
Unable to support delay differential equations.  The function 'delay(x, 1)' is not supported., at llvm::Value* rrllvm::ASTNodeCodeGen::delayExprCodeGen(const libsbml::ASTNode*)
Unable to support delay differential equations.  The function 'delay(x, temp)' is not supported., at llvm::Value* rrllvm::ASTNodeCodeGen::delayExprCodeGen(const libsbml::ASTNode*)
Unable to support delay differential equations.  The function 'delay(x, time / 2)' is not supported., at llvm::Value* rrllvm::ASTNodeCodeGen::delayExprCodeGen(const libsbml::ASTNode*)
Unknown ASTNode type of 323, at llvm::Value* rrllvm::ASTNodeCodeGen::codeGen(const libsbml::ASTNode*)
Unknown ASTNode type of 323, from <assignmentRule variable="P0">
Unknown ASTNode type of 323, from <assignmentRule variable="P1">
Unknown ASTNode type of 323, from <assignmentRule variable="p2">
Unknown ASTNode type of 323, from <assignmentRule variable="p3">
Unknown ASTNode type of 323, from <assignmentRule variable="x">
Unknown ASTNode type of 323, from <delay>
Unknown ASTNode type of 323, from <eventAssignment variable="P2">
Unknown ASTNode type of 323, from <eventAssignment variable="p2">
Unknown ASTNode type of 323, from <initialAssignment symbol="p2">
Unknown ASTNode type of 323, from <kineticLaw>
Unknown ASTNode type of 323, from <priority>
Unknown ASTNode type of 323, from <rateRule variable="p2">
reset

The errors excluding Unable to support algebraic rules

[pgleeson]

@robertvi As discussed in the meeting, the algebraic rules and delay differential equations are common reasons sbml simulators fail. Would it be possible to parse the error messages, and give a summary of the numbers of the top types of errors, and provide the detailed error when you click on the cell:

case	tellurium
cases=1690	fails=869 delay=100 algebraic=200 unknown=200
test123	pass
test124	fail (algebraic) Unable to support algebraic rules. The formula ...
test125	fail (delay) Unable to support delay differential equations. The function ...

[robertvi]

Note: no longer takes more than an hour to run, after I inserted command to close matplotlib plots between each test case (these were building up). Now takes a few minutes to run on my laptop in a linux VM.