Add remote links to results table and deal with engines that seem to pass but have the wrong results

stellaprins commented 1 week ago

[x] Add links as in foldout section of pass / FAIL (R) column

Amici, gillespy2, and vcell seem to pass while they do not really pass.

For remote runs:

[x] in logs find "simulation failed: Reached maximum number of steps" for AMICI and change colour coding to "❌ FAIL" depending on that.
[x] Investigate what goes wrong for gillespy2 and change colour coding depending on that (no specific reason found)
[x] investigate why vcell does not pass and change label to "❌ FAIL" (status: "QUEUED")

Do same for local runs:

[x] in logs find "simulation failed: Reached maximum number of steps" for AMICI and change colour coding to "❌ FAIL" depending on that.
[x] Investigate what goes wrong for gillespy2 and change colour coding depending on that (no specific reason found)
[x] investigate why vcell does not pass and change label to "❌ FAIL"

Address Padraig's comment on PR #69

[x] Check whether the message in "Error (R)" column refers to the outcome of the test in "pass / FAIL (R)". If so, put error message under expandable arrow under the XFAIL/FAIL message in the "Error (R)" column, not in separate columns.

stellaprins commented 3 days ago

Errors: remote runs

[x] Check whether log.yml files downloaded look the same as the log information found in the logs link (they are, see below)
[x] Check amici log.yml
[x] Check gillepsypy log.yml
[x] Check what information is extracted from where for the results table
[x] Move extracted info from current columns to collapsible content in the pass / fail column

the log.yml is imported as dictionary and if the status is FAILED the exception>message and exception>type are saved.

amici (pass)

amici logs info

log.yml downloaded from 'download' link: 'https://api.biosimulations.org/results/6720c0a5f9c43d155d41987f/download'

duration: 22.481979
exception: null
output: ''
sedDocuments:
- duration: 22.106298
  exception: null
  location: tmp566248
  output: "Archive contains 1 SED-ML documents with 1 models, 1 simulations, 1 tasks,\
    \ 1 reports, and 1 plots:\r\n  tmp566248:\r\n    Tasks (1):\r\n      sim1_net1\r\
    \n    Reports (1):\r\n      ex9: 3 data sets\r\n    Plots (1):\r\n      d1: 2\
    \ curves\r\n\r\nExecuting SED-ML file 1: tmp566248 ...\r\n  Found 1 tasks and\
    \ 2 outputs:\r\n    Tasks:\r\n      `sim1_net1`\r\n    Outputs:\r\n      `d1`\r\
    \n      `ex9`\r\n  Executing task 1: `sim1_net1`\r\n    Executing simulation ...\
    \ \e[34msucceeded\e[0m\r\n    Generating 2 outputs ...\r\n      Generating output\
    \ 1: `d1` ... \e[34msucceeded\e[0m\r\n      Generating output 2: `ex9` ..."
  outputs:
  - curves:
    - id: curve_V
      status: SUCCEEDED
    - id: curve_W
      status: SUCCEEDED
    duration: 0.922956
    exception: null
    id: d1
    output: ''
    skipReason: null
    status: SUCCEEDED
  - dataSets:
    - id: ex9_time
      status: SUCCEEDED
    - id: ex9_V
      status: SUCCEEDED
    - id: ex9_W
      status: SUCCEEDED
    duration: 0.332773
    exception: null
    id: ex9
    output: ''
    skipReason: null
    status: SUCCEEDED
  skipReason: null
  status: SUCCEEDED
  tasks:
  - algorithm: KISAO_0000496
    duration: 20.774802
    exception: null
    id: sim1_net1
    output: "2024-10-29 07:01:47.333 - amici.swig_wrappers - ERROR - [OTHER] AMICI\
      \ simulation failed: Reached maximum number of steps 10000 before reaching tout\
      \ at t=99.66.\r\n2024-10-29 07:01:47.333 - amici.swig_wrappers - DEBUG - [BACKTRACE]\
      \ The previous error occurred at:\r\n2       0x7ffff3462a8b /usr/local/lib/python3.11/site-packages/amici/_amici.cpython-311-x86_64-linux-gnu.so(+0xb5a8b)\
      \ [0x7ffff3462a8b]\r\n3       0x7ffff3556262 amici::Solver::run(double) const\
      \ + 34\r\n4       0x7ffff358e5d0 amici::ForwardProblem::workForwardProblem()\
      \ + 1728\r\n5       0x7ffff3542763 amici::runAmiciSimulation(amici::Solver&,\
      \ amici::ExpData const*, amici::Model&, bool) + 435\r\n6       0x7ffff34ec280\
      \ /usr/local/lib/python3.11/site-packages/amici/_amici.cpython-311-x86_64-linux-gnu.so(+0x13f\r\
      \n"
    simulatorDetails:
    - key: solver
      value: amici.amici.CVodeSolver
    - key: method
      value: amici.swig_wrappers.runAmiciSimulation
    - key: arguments
      value: {}
    skipReason: null
    status: SUCCEEDED
skipReason: null
status: SUCCEEDED

gillespy2

log.yml downloaded from 'download' link: 'https://api.biosimulations.org/results/67221c7d953bc3bf105ee735/download'

duration: 1.594634
exception: null
output: ''
sedDocuments:
- duration: 1.362146
  exception: null
  location: tmp950679
  output: "Archive contains 1 SED-ML documents with 1 models, 1 simulations, 1 tasks,\
    \ 1 reports, and 1 plots:\r\n  tmp950679:\r\n    Tasks (1):\r\n      sim1_net1\r\
    \n    Reports (1):\r\n      ex9: 3 data sets\r\n    Plots (1):\r\n      d1: 2\
    \ curves\r\n\r\nExecuting SED-ML file 1: tmp950679 ...\r\n  Found 1 tasks and\
    \ 2 outputs:\r\n    Tasks:\r\n      `sim1_net1`\r\n    Outputs:\r\n      `d1`\r\
    \n      `ex9`\r\n  Executing task 1: `sim1_net1`\r\n    Executing simulation ...\
    \ \e[34msucceeded\e[0m\r\n    Generating 2 outputs ...\r\n      Generating output\
    \ 1: `d1` ... \e[34msucceeded\e[0m\r\n      Generating output 2: `ex9` ..."
  outputs:
  - curves:
    - id: curve_V
      status: SUCCEEDED
    - id: curve_W
      status: SUCCEEDED
    duration: 0.773216
    exception: null
    id: d1
    output: ''
    skipReason: null
    status: SUCCEEDED
  - dataSets:
    - id: ex9_time
      status: SUCCEEDED
    - id: ex9_V
      status: SUCCEEDED
    - id: ex9_W
      status: SUCCEEDED
    duration: 0.360343
    exception: null
    id: ex9
    output: ''
    skipReason: null
    status: SUCCEEDED
  skipReason: null
  status: SUCCEEDED
  tasks:
  - algorithm: KISAO_0000088
    duration: 0.158526
    exception: null
    id: sim1_net1
    output: ''
    simulatorDetails:
    - key: method
      value: gillespy2.solvers.numpy.ode_solver.ODESolver
    - key: arguments
      value:
        integrator: lsoda
    skipReason: null
    status: SUCCEEDED
skipReason: null
status: SUCCEEDED

bionetgen (FAILED)

log.yml downloaded from 'download' link:'https://api.biosimulations.org/results/6720c0a5f9c43d155d41987f/download


duration: 0.585024
exception:
  message: "The COMBINE/OMEX did not execute successfully:\n\n  The SED document did\
    \ not execute successfully:\n  \n    Language for model `net1` is not supported.\n\
    \      - Model language `urn:sedml:language:sbml` is not supported. Models must\
    \ be in BNGL format (e.g., `sed:model/@language` must match `^urn:sedml:language:bngl(\\\
    .|$)` such as `urn:sedml:language:bngl`)."
  type: CombineArchiveExecutionError
output: ''
sedDocuments:
- duration: 0.333616
  exception:
    message: "The SED document did not execute successfully:\n\n  Language for model\
      \ `net1` is not supported.\n    - Model language `urn:sedml:language:sbml` is\
      \ not supported. Models must be in BNGL format (e.g., `sed:model/@language`\
      \ must match `^urn:sedml:language:bngl(\\.|$)` such as `urn:sedml:language:bngl`)."
    type: SedmlExecutionError
  location: tmp10738
  output: "Archive contains 1 SED-ML documents with 1 models, 1 simulations, 1 tasks,\
    \ 1 reports, and 1 plots:\r\n  tmp10738:\r\n    Tasks (1):\r\n      sim1_net1\r\
    \n    Reports (1):\r\n      ex9: 3 data sets\r\n    Plots (1):\r\n      d1: 2\
    \ curves\r\n\r\nExecuting SED-ML file 1: tmp10738 ...\r\n  Found 1 tasks and 2\
    \ outputs:\r\n    Tasks:\r\n      `sim1_net1`\r\n    Outputs:\r\n      `d1`\r\n\
    \      `ex9`\r\n  Executing task 1: `sim1_net1`\r\n    Executing simulation ...\
    \ \e[31mfailed\e[0m\r\n    Generating 2 outputs ...\r\n      Generating output\
    \ 1: `d1` ... \e[36mqueued\e[0m\r\n      Generating output 2: `ex9` ..."
  outputs:
  - curves:
    - id: curve_V
      status: SKIPPED
    - id: curve_W
      status: SKIPPED
    duration: 0.162714
    exception: null
    id: d1
    output: ''
    skipReason: null
    status: SKIPPED
  - dataSets:
    - id: ex9_time
      status: SKIPPED
    - id: ex9_V
      status: SKIPPED
    - id: ex9_W
      status: SKIPPED
    duration: 0.059287
    exception: null
    id: ex9
    output: ''
    skipReason: null
    status: SKIPPED
  skipReason: null
  status: FAILED
  tasks:
  - algorithm: null
    duration: 0.026
    exception:
      message: "Language for model `net1` is not supported.\n  - Model language `urn:sedml:language:sbml`\
        \ is not supported. Models must be in BNGL format (e.g., `sed:model/@language`\
        \ must match `^urn:sedml:language:bngl(\\.|$)` such as `urn:sedml:language:bngl`)."
      type: ValueError
    id: sim1_net1
    output: ''
    simulatorDetails: null
    skipReason: null
    status: FAILED
skipReason: null
status: FAILED

stellaprins commented 3 days ago

H5 files: remote runs

Nothing is currently done with the h5 files containing the information related to the d1 plots

amici (pass)

Can you, by tweeking the SED-ML file, export a csv instead of an HDF5 file?

stellaprins commented 2 days ago

gillespy2

Issue: seems to pass but faulty d1 plot.

Search for warning / error message in

[x] check log.yml (no obvious error / warning signs)
[x] check 'logs' from link below ("failed" mentioned in outputs--> compare to a "true pass")
[x] check 'view' from link below (no obvious error / warning signs)
[x] check for word "stochastic" (it is not in there, but we think that the false pass is because gillespy2 is designed to run stochastic models)

Check whether "failed" in logs reveals a reason behind the fail

[x] Compare logs link from Gillpsypy (false pass) with Copasi (true pass) ("failed also mentioned in outputs)

gillespy2 view: https://api.biosimulations.org/runs/67221c7d953bc3bf105ee735 download: https://api.biosimulations.org/results/67221c7d953bc3bf105ee735/download logs: https://api.biosimulations.org/logs/67221c7d953bc3bf105ee735?includeOutput=true

Copasi View: https://api.biosimulations.org/runs/672220b22e0c0591a7c8879b Downloads: https://api.biosimulations.org/results/672220b22e0c0591a7c8879b/download Logs: https://api.biosimulations.org/logs/672220b22e0c0591a7c8879b?includeOutput=true

stellaprins commented 10 hours ago

vcell

It seems that there is a RuntimeException (which can be a lot of things).

[ ] Mark cases where detailedErrorLog.txt is created with "RuntimeException" as FAIL.

Remote

View: https://api.biosimulations.org/runs/6724ce8d0d09353e8f0f48cf Downloads: https://api.biosimulations.org/results/6724ce8d0d09353e8f0f48cf/download Logs: https://api.biosimulations.org/logs/6724ce8d0d09353e8f0f48cf?includeOutput=true

log.yml

duration: null
exception: null
output: null
sedDocuments: []
skipReason: null
status: QUEUED

Local

log.yml

duration: 4
exception: null
output: Processing tmp456126.  Done
sedDocuments: []
skipReason: null
status: SUCCEEDED

for vcell there are logs that I haven't seen before for any of the other engines: detailedErrorLog.txt

LEMS_NML2_Ex9_FN_missing_xmlns4736092972325131743,  doc:    RuntimeException: SED-ML processing for /tmp/vcell_temp_df395b63-9d48-474e-8aeb-d9ab9188bf8f12924121859794883090/tmp882500 failed with error: Failed to create file /root/out/temp/simulation_tmp882500
LEMS_NML2_Ex9_FN_missing_xmlns4736092972325131743,  doc:    NullPointerException: Cannot invoke "org.jlibsedml.SedML.getPathForURI()" because "this.sedml" is nullCannot invoke "org.jlibsedml.SedML.getPathForURI()" because "sedml" is null
LEMS_NML2_Ex9_FN_missing_xmlns4736092972325131743,  doc:    RuntimeException: Cannot invoke "org.jlibsedml.SedML.getPathForURI()" because "this.sedml" is nullCannot invoke "org.jlibsedml.SedML.getPathForURI()" because "sedml" is nullFailure executing the sed document.

detailedResultLog.txt

BaseName,Models,Sims,Tasks,Outputs,BioModels,HasOverrides,HasScans,NumSimsSuccessful
LEMS_NML2_Ex9_FN_missing_xmlns4736092972325131743,tmp882500,Failed to create file /root/out/temp/simulation_tmp882500
LEMS_NML2_Ex9_FN_missing_xmlns4736092972325131743,1,1,1,2,0,false,false,0
LEMS_NML2_Ex9_FN_missing_xmlns4736092972325131743,1,1,1,2,0,false,false,0

fullSuccessLog.txt LEMS_NML2_Ex9_FN_missing_xmlns4736092972325131743

OpenSourceBrain / SBMLShowcase