pynml-archive: add `-sedml`

sanjayankur31 commented 2 months ago

@pgleeson : do we have a different format string for LEMS? I can't find one. This is what the manifest current looks like:

<?xml version="1.0" encoding="utf-8"?>
<omexManifest xmlns="http://identifiers.org/combine.specifications/omex-manifest">
        <content location="." format="http://identifiers.org/combine.specifications/omex"/>
        <content location="LEMS_NML2_Ex5_DetCell.sedml" master="true" format="http://identifiers.org/combine.specifications/sed-ml"/>
        <content location="LEMS_NML2_Ex5_DetCell.xml"  format="http://identifiers.org/combine.specifications/neuroml"/>
        <content location="NaConductance.channel.nml"  format="http://identifiers.org/combine.specifications/neuroml"/>
        <content location="KConductance.channel.nml"  format="http://identifiers.org/combine.specifications/neuroml"/>
        <content location="LeakConductance.channel.nml"  format="http://identifiers.org/combine.specifications/neuroml"/>
        <content location="NML2_SingleCompHHCell.nml"  format="http://identifiers.org/combine.specifications/neuroml"/>
</omexManifest>

pgleeson commented 2 months ago

Not sure... might be best to run an example model on the biosim website and then try to down load the omex file...
What I noticed was it didn't like *neux files, and also seemed to want a metadata file in there too, with rdf. Not sure what goes in that that's not in the manifest...

sanjayankur31 commented 2 months ago

I'll try downloading an example to see what they do.

In the meantime, I've tweaked it so that if the master file is sedml, the archive will be *.omex. Will check what the rdf file includes---that isn't implemented at the moment.

sanjayankur31 commented 2 months ago

I downloaded this, extracted it, created an archive and re-uploaded it:

https://biosimulations.org/projects/Hodgkin-Huxley-cell-Gleeson

It errored with (formatted):

"`/root/archive.omex` is not a valid COMBINE/OMEX archive.
  - The SED-ML file at location `LEMS_model.sedml` is invalid.
    - The value of a 'sedml:id' must conform to the syntax of the <sbml> data type 'SId'
      Reference: L3V1 Sedml V1 Section
       The id on the <report> is 'results/ex5_v', which does not conform to the syntax.

    - The value of a 'sedml:id' must conform to the syntax of the <sbml> data type 'SId'
      Reference: L3V1 Sedml V1 Section
       The id on the <report> is 'results/ex5_vars', which does not conform to the syntax.

So, the id attribute cannot have / in it from the looks of it. We'll need to replace it with _ or something in org.neuroml.export. Full output (formatted):

  "output": "\u001b[0;36mThank you for using runBioSimulations!\u001b[0m

\u001b[0;36m================================================ Loading Singularity ================================================\u001b[0m

\u001b[0;36m=================================================== Set up storage ==================================================\u001b[0m

\u001b[0;36m========================================== Downloading COMBINE/OMEX archive =========================================\u001b[0m
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r100   139  100   139    0     0    346      0 --:--:-- --:--:-- --:--:--   345
\r100 23446  100 23446    0     0  43446      0 --:--:-- --:--:-- --:--:-- 43446

\u001b[0;36m=========================================== Executing COMBINE/OMEX archive ==========================================\u001b[0m
INFO:    Using cached SIF image
\u001b[31m`/root/archive.omex` is not a valid COMBINE/OMEX archive.
  - The SED-ML file at location `LEMS_model.sedml` is invalid.
    - The value of a 'sedml:id' must conform to the syntax of the <sbml> data type 'SId'
      Reference: L3V1 Sedml V1 Section
       The id on the <report> is 'results/ex5_v', which does not conform to the syntax.

    - The value of a 'sedml:id' must conform to the syntax of the <sbml> data type 'SId'
      Reference: L3V1 Sedml V1 Section
       The id on the <report> is 'results/ex5_vars', which does not conform to the syntax.
      \u001b[0m
srun: error: xanadu-69: task 0: Exited with exit code 1
srun: Terminating job step 7915253.1

\u001b[0;36m===================================================== Saving log ====================================================\u001b[0m
Completed 1.8 KiB/1.8 KiB (11.7 KiB/s) with 1 file(s) remaining\rupload: ./rawLog.txt to s3://files.biosimulations.org/simulations/661ff1f2d1633ac5a9764025/rawLog.txt

\u001b[0;36m===================================================== Saving Structured log ==========================================\u001b[0m
Completed 750 Bytes/750 Bytes (6.0 KiB/s) with 1 file(s) remaining\rupload: outputs/log.yml to s3://files.biosimulations.org/simulations/661ff1f2d1633ac5a9764025/outputs/log.yml

\u001b[0;36m================================================== Zipping outputs ==================================================\u001b[0m
  adding: outputs/ (stored 0%)
  adding: outputs/log.yml (deflated 52%)
  adding: rawLog.txt (deflated 70%)

\u001b[0;36m=================================================== Saving HDF5 outputs ==================================================\u001b[0m

\u001b[0;36m=================================================== Saving non-HDF5 outputs ==================================================\u001b[0m
Completed 1.6 KiB/4.4 KiB (15.3 KiB/s) with 2 file(s) remaining\rupload: ./output.zip to s3://files.biosimulations.org/simulations/661ff1f2d1633ac5a9764025/output.zip
Completed 1.6 KiB/4.4 KiB (15.3 KiB/s) with 1 file(s) remaining\rCompleted 4.4 KiB/4.4 KiB (28.2 KiB/s) with 1 file(s) remaining\rupload: ./rawLog.txt to s3://files.biosimulations.org/simulations/661ff1f2d1633ac5a9764025/rawLog.txt

\u001b[0;36m====================================== Unzip contents of COMBINE/OMEX archive ======================================\u001b[0m
Archive:  archive.omex
 extracting: contents/LEMS_model.sedml  
 extracting: contents/LEMS_model.xml  
 extracting: contents/NaConductance.channel.nml  
 extracting: contents/KConductance.channel.nml  
 extracting: contents/LeakConductance.channel.nml  
 extracting: contents/NML2_SingleCompHHCell.nml  
 extracting: contents/manifest.xml   

\u001b[0;36m====================================== Save contents of COMBINE/OMEX archive ======================================\u001b[0m
Completed 10.4 KiB/22.1 KiB (88.6 KiB/s) with 7 file(s) remaining\rupload: contents/LEMS_model.sedml to s3://files.biosimulations.org/simulations/661ff1f2d1633ac5a9764025/contents/LEMS_model.sedml
Completed 10.4 KiB/22.1 KiB (88.6 KiB/s) with 6 file(s) remaining\rCompleted 12.5 KiB/22.1 KiB (82.1 KiB/s) with 6 file(s) remaining\rupload: contents/KConductance.channel.nml to s3://files.biosimulations.org/simulations/661ff1f2d1633ac5a9764025/contents/KConductance.channel.nml
Completed 12.5 KiB/22.1 KiB (82.1 KiB/s) with 5 file(s) remaining\rCompleted 13.1 KiB/22.1 KiB (84.4 KiB/s) with 5 file(s) remaining\rupload: contents/LeakConductance.channel.nml to s3://files.biosimulations.org/simulations/661ff1f2d1633ac5a9764025/contents/LeakConductance.channel.nml
Completed 13.1 KiB/22.1 KiB (84.4 KiB/s) with 4 file(s) remaining\rCompleted 16.5 KiB/22.1 KiB (103.0 KiB/s) with 4 file(s) remaining\rupload: contents/LEMS_model.xml to s3://files.biosimulations.org/simulations/661ff1f2d1633ac5a9764025/contents/LEMS_model.xml
Completed 16.5 KiB/22.1 KiB (103.0 KiB/s) with 3 file(s) remaining\rCompleted 18.8 KiB/22.1 KiB (115.9 KiB/s) with 3 file(s) remaining\rupload: contents/NML2_SingleCompHHCell.nml to s3://files.biosimulations.org/simulations/661ff1f2d1633ac5a9764025/contents/NML2_SingleCompHHCell.nml
Completed 18.8 KiB/22.1 KiB (115.9 KiB/s) with 2 file(s) remaining\rCompleted 19.7 KiB/22.1 KiB (102.1 KiB/s) with 2 file(s) remaining\rupload: contents/manifest.xml to s3://files.biosimulations.org/simulations/661ff1f2d1633ac5a9764025/contents/manifest.xml
Completed 19.7 KiB/22.1 KiB (102.1 KiB/s) with 1 file(s) remaining\rCompleted 22.1 KiB/22.1 KiB (112.0 KiB/s) with 1 file(s) remaining\rupload: contents/NaConductance.channel.nml to s3://files.biosimulations.org/simulations/661ff1f2d1633ac5a9764025/contents/NaConductance.channel.nml

\u001b[0;36m==================================================== Updating log (2) ===============================================\u001b[0m
Completed 5.7 KiB/5.7 KiB (43.8 KiB/s) with 1 file(s) remaining\rupload: ./rawLog.txt to s3://files.biosimulations.org/simulations/661ff1f2d1633ac5a9764025/rawLog.txt

\u001b[0;36m================================================== Saving final log =================================================\u001b[0m

\u001b[0;36m=========================================== Post-processing simulation run ==========================================\u001b[0m
\u001b[0;32mProcess the manifest file in the COMBINE archive ... succeeded.
Read the manifest and post the files to the API ... succeeded.
Create thumbnails from the images specified in the manifest ... succeeded.
Post thumbnails to the API ... succeeded.
Process the SED-ML file in the COMBINE archive ... succeeded.
Post the SED-ML file to the API ... succeeded.
Retrieve the logs of the simulation run ... succeeded.
Post the logs to the API ... succeeded.
Process and upload the outputs of the simulation run ... succeeded.\u001b[0m
\u001b[1;33mProcess the metadata in the COMBINE archive ... failed due to: The metadata of the file could not be found:
There was an error in processing the metadata file. More information is available at https://docs.biosimulations.org/concepts/conventions/simulation-project-metadata/. A validation tool is available at https://run.biosimulations.org/utils/validate-metadata.
Post the metadata to the API ... failed due to: Metadata not found due to a dependent step failing.\u001b[0m
\u001b[0;36m================================ Run complete. Thank you for using runBioSimulations! ===============================\u001b[0m",

So it does require some metadata also. Will have to figure out how that works.

sanjayankur31 commented 2 months ago

Also verified with pynml -validate-sedml now:

$ pynml -validate-sedm LEMS_model.sedml
line 181: (10302 [Error]) The value of a 'sedml:id' must conform to the syntax of the <sbml> data type 'SId'
Reference: L3V1 Sedml V1 Section
 The id on the <report> is 'results/ex5_v', which does not conform to the syntax.

line 187: (10302 [Error]) The value of a 'sedml:id' must conform to the syntax of the <sbml> data type 'SId'
Reference: L3V1 Sedml V1 Section
 The id on the <report> is 'results/ex5_vars', which does not conform to the syntax.

pyNeuroML >>> ERROR - one or more SEDML files failed to validate

NeuroML / pyNeuroML

pynml-archive: add `-sedml` #335