search

the-virtual-brain / tvb-recon

Tools for creating full TVB models from individual anatomical scans
Other
34 stars 21 forks source link

Missing config file errors #55

Closed RdoubleA closed 4 years ago

RdoubleA commented 6 years ago

Hey all,

This is my first time using tvb-recon. I have my dataset directory structure organized as described in the readme. I run the docker command like so:

docker run -it -v /Users/rafi/Documents/Stanford/Saggar_Lab/HPI_Reflect/TVB_patients/:/home/submitter/data -v /Users/rafi/Documents/Stanford/Saggar_Lab/tvb-recon/:/opt/tvb-recon thevirtualbrain/tvb-recon /bin/bash

Yet after I run sudo condor_master and cd into pegasus and run run_sequentially.py, I get the following error:

Starting to process the following subjects: %s [1]
Starting to process the subject: TVB1
Traceback (most recent call last):
  File "run_sequentially.py", line 172, in <module>
    prepare_config_for_new_atlas(current_dir, atlas)
  File "run_sequentially.py", line 97, in prepare_config_for_new_atlas
    with open(current_patient_props_path, "r") as current_patient_props_file:
IOError: [Errno 2] No such file or directory: '/home/submitter/data/TVB1/configs/patient_flow.properties'

I thought the pipeline would automatically create the default configuration files from what I understood. Then I tried copying the configuration files from the pegasus/configs folder into the TVB1/configs folder. I reran and I'm still getting missing file errors.

Starting to process the following subjects: %s [1]
Starting to process the subject: TVB1
Configured atlas default for patient inside folder /home/submitter/data/TVB1/configs
Checking currently running job ids...
Currently running job ids are: []
Starting pegasus run for subject: TVB1with atlas: default
main_pegasus.sh: 7: main_pegasus.sh: Bad substitution
/opt/tvb-recon
2018-08-29 23:01:39,473 - tvb.recon.dax.configuration - INFO - Parsing patient configuration file /home/submitter/data/TVB1/configs/patient_flow.properties
Traceback (most recent call last):
  File "/opt/conda/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/opt/conda/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/opt/tvb-recon/tvb/recon/dax/__main__.py", line 81, in <module>
    job_aparc_aseg_in_d, job_mapping_details)
  File "tvb/recon/dax/tracts_generation.py", line 86, in add_tracts_generation_steps
    gm_mif = File(DWIFiles.GM_MIF.value)
  File "/usr/lib/python2.7/dist-packages/enum/__init__.py", line 390, in __getattr__
    raise AttributeError(name)
AttributeError: GM_MIF
Traceback (most recent call last):
  File "/usr/bin/pegasus-graphviz", line 507, in <module>
    main()
  File "/usr/bin/pegasus-graphviz", line 497, in main
    dag = parse_daxfile(dagfile, options.files)
  File "/usr/bin/pegasus-graphviz", line 225, in parse_daxfile
    f = open(fname,"r")
IOError: [Errno 2] No such file or directory: '/home/submitter/data/TVB1/configs/dax/main_bnm.dax'
Error: dot: can't open /home/submitter/data/TVB1/configs/dax/main_bnm.dot
2018.08.29 23:01:39.741 UTC: [FATAL ERROR]
 [1]: Unable to instantiate Site Catalog  at edu.isi.pegasus.planner.catalog.site.SiteFactory.loadInstance(SiteFactory.java:234)
 [2]: edu.isi.pegasus.planner.catalog.site.impl.XML caught edu.isi.pegasus.planner.catalog.site.SiteCatalogException Cannot read or access file $path/sites.xml at edu.isi.pegasus.planner.catalog.site.impl.XML.connect(XML.java:146)
ERROR while logging metrics The metrics file location is not yet initialized
Checking currently running job ids...
Currently running job ids are: []
Traceback (most recent call last):
  File "run_sequentially.py", line 185, in <module>
    current_job_id = new_job_ids[0]
IndexError: list index out of range

I'm not really sure what to do at this point and would really appreciate any help!

popaula937 commented 6 years ago

Hi @RdoubleA,

Thanks for your interest in tvb-recon!

So, for the second part you have described, I just want to point out that the configuration files under pegasus/config are not final, so just copying them under TVB1/configs will not be enough. Regarding the "GM_MIF" AttributeError: this should happen only if inside the "patient_flow.properties", the attribute dwi.multi.shell=True. Have you changed that?

Secondly, as you said, the configuration files should be generated automatically by the pipeline. We have to identify why you have this problem. Is it possible that you have created a "configs" folder under TVB1 before running the pipeline? Can you remove the "configs" from TVB1 and retry the run?

RdoubleA commented 6 years ago

Hi, thank you for your quick response!

Yes, I manually changed that option when I copied the config files from pegasus.

I am no trying to run by removing the configs folder entirely. I'm still not sure what files it's expecting, because I run into this error:


Starting to process the following subjects: %s [1]
Starting to process the subject: TVB1
Folder /home/submitter/data/TVB1/configs has been created...
Configuration files for subject TVB1 are ready!
Checking currently running job ids...
Currently running job ids are: []
Starting pegasus run for subject: TVB1with atlas: default
main_pegasus.sh: 7: main_pegasus.sh: Bad substitution
/opt/tvb-recon
2018-08-30 19:39:59,573 - tvb.recon.dax.configuration - INFO - Parsing patient configuration file /home/submitter/data/TVB1/configs/patient_flow.properties
Traceback (most recent call last):
  File "/opt/conda/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/opt/conda/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/opt/tvb-recon/tvb/recon/dax/__main__.py", line 81, in <module>
    job_aparc_aseg_in_d, job_mapping_details)
  File "tvb/recon/dax/tracts_generation.py", line 86, in add_tracts_generation_steps
    gm_mif = File(DWIFiles.GM_MIF.value)
  File "/usr/lib/python2.7/dist-packages/enum/__init__.py", line 390, in __getattr__
    raise AttributeError(name)
AttributeError: GM_MIF
Traceback (most recent call last):
  File "/usr/bin/pegasus-graphviz", line 507, in <module>
    main()
  File "/usr/bin/pegasus-graphviz", line 497, in main
    dag = parse_daxfile(dagfile, options.files)
  File "/usr/bin/pegasus-graphviz", line 225, in parse_daxfile
    f = open(fname,"r")
IOError: [Errno 2] No such file or directory: '/home/submitter/data/TVB1/configs/dax/main_bnm.dax'
Error: dot: can't open /home/submitter/data/TVB1/configs/dax/main_bnm.dot
2018.08.30 19:40:00.843 UTC: [ERROR]  Problem while determining the version of dax class java.lang.RuntimeException: java.io.FileNotFoundException: The file (/home/submitter/data/TVB1/configs/dax/main_bnm.dax ) specified does not exist
2018.08.30 19:40:00.850 UTC: [FATAL ERROR]
 [1]: Instantiating DAXParser  at edu.isi.pegasus.planner.parser.DAXParserFactory.loadDAXParser(DAXParserFactory.java:235)
 [2]: Invalid static initializer method name for DAXParser3 at edu.isi.pegasus.common.util.DynamicLoader.instantiate(DynamicLoader.java:131)
ERROR while logging metrics The metrics file location is not yet initialized
Checking currently running job ids...
Currently running job ids are: []
Traceback (most recent call last):
  File "run_sequentially.py", line 185, in <module>
    current_job_id = new_job_ids[0]
IndexError: list index out of range
popaula937 commented 6 years ago

So, it seems that the configuration files have been generated now. You should see all the files from pegasus/config being copied under TVB1/configs, but some of the values are substituted.

I will have to ask you to put back to false the dwi.multi.shell=False because it will take the pipeline through a flow that we have not tested for a while and you might face this kind of errors..sorry for that, we will try to fix it soon (and thanks for finding the bug). It would be best to try at first with the default configurations. The dwi.multi.shell=False has to be changed inside TVB1/configs/patient_flow.properties, but it would be good to do it also under pegasus/config/patient_flow.properties for further runs.

Let me know how it goes.

RdoubleA commented 6 years ago

Okay it seems to have run through the entire pipeline, but it never generated any output files. I don't have an outputs folder in my TVB1 folder.

Starting to process the following subjects: %s [1]
Starting to process the subject: TVB1
Configured atlas default for patient inside folder /home/submitter/data/TVB1/configs
Checking currently running job ids...
Currently running job ids are: []
Starting pegasus run for subject: TVB1with atlas: default
main_pegasus.sh: 7: main_pegasus.sh: Bad substitution
/opt/tvb-recon
2018-08-30 21:51:49,678 - tvb.recon.dax.configuration - INFO - Parsing patient configuration file /home/submitter/data/TVB1/configs/patient_flow.properties
Removing Generate 5tt MIF -> Tracts SIFT
Removing gen_mapping_details -> convert_output
Removing Convert APARC+ASEG to NIFTI with good orientation -> convert_output
Removing Recon-all for T1 -> qc_snapshot
Removing Recon-all for T1 -> qc_snapshot
2018.08.30 21:51:53.459 UTC:
2018.08.30 21:51:53.465 UTC:   -----------------------------------------------------------------------
2018.08.30 21:51:53.471 UTC:   File for submitting this DAG to Condor           : TVB-PIPELINE-0.dag.condor.sub
2018.08.30 21:51:53.477 UTC:   Log of DAGMan debugging messages                 : TVB-PIPELINE-0.dag.dagman.out
2018.08.30 21:51:53.483 UTC:   Log of Condor library output                     : TVB-PIPELINE-0.dag.lib.out
2018.08.30 21:51:53.488 UTC:   Log of Condor library error messages             : TVB-PIPELINE-0.dag.lib.err
2018.08.30 21:51:53.495 UTC:   Log of the life of condor_dagman itself          : TVB-PIPELINE-0.dag.dagman.log
2018.08.30 21:51:53.500 UTC:
2018.08.30 21:51:53.518 UTC:   -----------------------------------------------------------------------
2018.08.30 21:51:54.624 UTC:   Created Pegasus database in: sqlite:////home/submitter/.pegasus/workflow.db
2018.08.30 21:51:54.630 UTC:   Your database is compatible with Pegasus version: 4.8.2
2018.08.30 21:51:54.707 UTC:   Submitting to condor TVB-PIPELINE-0.dag.condor.sub
2018.08.30 21:51:54.745 UTC:   Submitting job(s).
2018.08.30 21:51:54.753 UTC:   1 job(s) submitted to cluster 1.
2018.08.30 21:51:54.760 UTC:
2018.08.30 21:51:54.773 UTC:   Your workflow has been started and is running in the base directory:
2018.08.30 21:51:54.779 UTC:
2018.08.30 21:51:54.785 UTC:     /home/submitter/pegasus/submit/submitter/pegasus/TVB-PIPELINE/run0001
2018.08.30 21:51:54.793 UTC:
2018.08.30 21:51:54.805 UTC:   *** To monitor the workflow you can run ***
2018.08.30 21:51:54.813 UTC:
2018.08.30 21:51:54.826 UTC:     pegasus-status -l /home/submitter/pegasus/submit/submitter/pegasus/TVB-PIPELINE/run0001
2018.08.30 21:51:54.832 UTC:
2018.08.30 21:51:54.841 UTC:   *** To remove your workflow run ***
2018.08.30 21:51:54.847 UTC:
2018.08.30 21:51:54.858 UTC:     pegasus-remove /home/submitter/pegasus/submit/submitter/pegasus/TVB-PIPELINE/run0001
2018.08.30 21:51:54.866 UTC:
2018.08.30 21:51:55.994 UTC:   Time taken to execute is 3.857 seconds
Checking currently running job ids...
Currently running job ids are: ['0001']
The job that has been started has the id: 0001
Starting to monitor the submit folder: /home/submitter/pegasus/submit/submitter/pegasus/TVB-PIPELINE/run0001 ...
Checked at Thu, 30 Aug 2018 21:51:56 and monitord.done file was not generated yet!
The run has finished for job with id: 0001
Configured atlas a2009s for patient inside folder /home/submitter/data/TVB1/configs
Checking currently running job ids...
Currently running job ids are: []
Starting pegasus run for subject: TVB1with atlas: a2009s
main_pegasus.sh: 7: main_pegasus.sh: Bad substitution
/opt/tvb-recon
2018-08-30 22:01:56,733 - tvb.recon.dax.configuration - INFO - Parsing patient configuration file /home/submitter/data/TVB1/configs/patient_flow.properties
Removing Generate 5tt MIF -> Tracts SIFT
Removing gen_mapping_details -> convert_output
Removing Convert APARC+ASEG to NIFTI with good orientation -> convert_output
Removing Recon-all for T1 -> qc_snapshot
Removing Recon-all for T1 -> qc_snapshot
2018.08.30 22:01:59.445 UTC:
2018.08.30 22:01:59.451 UTC:   -----------------------------------------------------------------------
2018.08.30 22:01:59.457 UTC:   File for submitting this DAG to Condor           : TVB-PIPELINE-0.dag.condor.sub
2018.08.30 22:01:59.463 UTC:   Log of DAGMan debugging messages                 : TVB-PIPELINE-0.dag.dagman.out
2018.08.30 22:01:59.468 UTC:   Log of Condor library output                     : TVB-PIPELINE-0.dag.lib.out
2018.08.30 22:01:59.474 UTC:   Log of Condor library error messages             : TVB-PIPELINE-0.dag.lib.err
2018.08.30 22:01:59.480 UTC:   Log of the life of condor_dagman itself          : TVB-PIPELINE-0.dag.dagman.log
2018.08.30 22:01:59.487 UTC:
2018.08.30 22:01:59.505 UTC:   -----------------------------------------------------------------------
2018.08.30 22:02:00.236 UTC:   Your database is compatible with Pegasus version: 4.8.2
2018.08.30 22:02:00.318 UTC:   Submitting to condor TVB-PIPELINE-0.dag.condor.sub
2018.08.30 22:02:00.345 UTC:   Submitting job(s).
2018.08.30 22:02:00.353 UTC:   1 job(s) submitted to cluster 17.
2018.08.30 22:02:00.360 UTC:
2018.08.30 22:02:00.366 UTC:   Your workflow has been started and is running in the base directory:
2018.08.30 22:02:00.372 UTC:
2018.08.30 22:02:00.379 UTC:     /home/submitter/pegasus/submit/submitter/pegasus/TVB-PIPELINE/run0002
2018.08.30 22:02:00.384 UTC:
2018.08.30 22:02:00.391 UTC:   *** To monitor the workflow you can run ***
2018.08.30 22:02:00.397 UTC:
2018.08.30 22:02:00.403 UTC:     pegasus-status -l /home/submitter/pegasus/submit/submitter/pegasus/TVB-PIPELINE/run0002
2018.08.30 22:02:00.410 UTC:
2018.08.30 22:02:00.418 UTC:   *** To remove your workflow run ***
2018.08.30 22:02:00.424 UTC:
2018.08.30 22:02:00.431 UTC:     pegasus-remove /home/submitter/pegasus/submit/submitter/pegasus/TVB-PIPELINE/run0002
2018.08.30 22:02:00.437 UTC:
2018.08.30 22:02:01.060 UTC:   Time taken to execute is 2.698 seconds
Checking currently running job ids...
Currently running job ids are: ['0002']
The job that has been started has the id: 0002
Starting to monitor the submit folder: /home/submitter/pegasus/submit/submitter/pegasus/TVB-PIPELINE/run0002 ...
Checked at Thu, 30 Aug 2018 22:02:01 and monitord.done file was not generated yet!
The run has finished for job with id: 0002
popaula937 commented 6 years ago

It seems there were some errors in the workflow, at an early stage. In order to debug the problem, in the same docker container, you can use: pegasus-status -l /home/submitter/pegasus/submit/submitter/pegasus/TVB-PIPELINE/run0001 and: pegasus-analyzer /home/submitter/pegasus/submit/submitter/pegasus/TVB-PIPELINE/run0001

If you want to check the status while the flow is running, you will have to open a new terminal and bash inside the same docker container with: docker exec -i -t container_id /bin/bash

liadomide commented 4 years ago

I close this, due to lack of response. I assume it is something local