usability questions

[mihai-sysbio]

Thanks for putting together these test suites, very nice work! I have some usability questions (possibly issues):

A) Minor: it's unclear what model formats are accepted, since it's just a string of the model file. The example points to json, which is not what I would have expected.

B) Minor: it's unclear to me why there is the need to have 3 different docker images instead of one. As a user, I'd prefer to have one image that includes all everything (including dependencies), and pass different commands to it with docker run.

C) Major: I can't seem to run the Memote-style test on yeast-GEM:

FBCModelTests.Memote.generate_report("yeast-GEM.mat", Tulip.Optimizer)

since it ends up in an infinite loop that prints:

^C┌ Warning: Increase regularizations to 1.4901161193847656e-6
└ @ Tulip ~/.julia/packages/Tulip/SRdil/src/IPM/HSD/step.jl:47

[stelmo]

Ah, thanks for finding these issues! I will look into C this week and get back to you.

[stelmo]

Actually, I have put time aside for this at the end of next week

[exaexa]

Hi, just a note re. B) with the docker images -- I wanted to separate off the FROG part that does only comparisons (tiny docker) from the one that also has to bundle the solvers and generate the reports. This is subject to change/simplify (I'll probably make 1 generic docker and have the other ones only as install/entrypoint patches over that).

The solver-in-docker problem is not a nice one though; I'd love to be able to somehow get gurobi licenses in there :]

[mihai-sysbio]

The solver-in-docker problem is not a nice one though; I'd love to be able to somehow get gurobi licenses in there :]

How about mounting the license file at the expected location?

[exaexa]

yeah that should work, I really just need to find a moment try it (meaning I need to get the license etc. :D )

[mihai-sysbio]

This might come in handy, although it's valid only for a short window:

[stelmo]

@mihai-sysbio, I think most of your issues actually came from the optimizer you were using, Tulip.jl. The latest release of FBCModelTests, v0.2.5, changes the way results are collected, and which tests are run. It should be more robust. We also changed the default solver to GLPK. Next week we will release v.0.3.0, which adds a few more SBO tests and makes some cosmetic changes. I tested your model, using Gurobi as the solver, and the results are shown below.

Note, we do not yet handle split objective functions (see issue #27), so your model will fail all the tests that rely on identifying the objective function. Finally, we only use SBO annotations to identify reaction types, which might be another source of problems. Let me know if you have any questions!

Edit: we will soon release v0.3.0, which improves some details of the tests, I have used this branch (#76) to generate the results below.

[mihai-sysbio]

Next week we will release v.0.3.0 Edit: we will soon release v0.3.0

Is there any news regarding the upcoming release?

[exaexa]

Hi, ok whoops, looks like someone (me) forgot to pull the trigger. Will release the 0.3 now :] -mk

[exaexa]

(done)