search

opencobra / memote

memote – the genome-scale metabolic model test suite
https://memote.readthedocs.io/
Apache License 2.0
128 stars 28 forks source link

SBO statistics calculated incorrectly #626

Open matthiaskoenig opened 5 years ago

matthiaskoenig commented 5 years ago

SBO statistics are not calculated correctly. In the attached screenshot and model example I have either 100% SBO, skipped SBO or errored SBO. I.e. on everything which works I have 100%. How does this result in 36% SBO? Also there should be exchange reactions in the model after loading, not sure why memote is erroring here (this seems to be a bug)

sbo_example

tiny_example_9.xml.zip

Model can be loaded with the cobrapy libsbml branch https://github.com/opencobra/cobrapy/tree/libsbml

Midnighter commented 5 years ago

In the attached screenshot and model example I have either 100% SBO, skipped SBO or errored SBO. I.e. on everything which works I have 100%. How does this result in 36% SBO?

Skipped or error tests are counted as 0%. That means that in the SBO section you get 4 / 11 ~ 36% so I don't really see the problem there?

Also there should be exchange reactions in the model after loading, not sure why memote is erroring here (this seems to be a bug)

With the libsbml branch this didn't error for me but the assertion is still wrong and the message is formulated confusingly. We'll make a fix.

matthiaskoenig commented 5 years ago

I already submitted a bugfix to the libsbml branch which fixed the errors. That is why you did not see the errors any more.

JonathanRob commented 5 years ago

This might be a matter of opinion, but I agree with @matthiaskoenig that the calculation of the overall score could use another look - at least with regard to some of the "Skipped" tests. For example, if I'm evaluating a model that doesn't contain any sink or demand reactions, the maximum score I can achieve here is ~82%.

This implies that a model should contain sink and demand reactions, which seems a bit odd to me.

Midnighter commented 5 years ago

This implies that a model should contain sink and demand reactions, which seems a bit odd to me.

Yes, that shouldn't be the case, of course.