pipeline

[riccardomurri]

From @SaadiaIftikhar on June 2, 2016 13:10

I tried various possibilities, but the error remained the same for the pipeline:

() | _ _ | | tmlib.workflow.jterator (0.0.7) | | / -) '/ ` | / \ '| Image analysis pipeline engine / |**|| **,|/_| https://github.com/TissueMAPS/TmLibrary |/

2016-06-02 13:07:14 | INFO | tmlib.workflow.cli | delete previous job output 2016-06-02 13:07:14 | INFO | tmlib.workflow.jterator.api | delete existing mapobject types 2016-06-02 13:07:14 | INFO | tmlib.workflow.cli | create batches The following error occurred while reading from file: while parsing a block mapping in "", line 2, column 1: input: ^ expected , but found '-' in "", line 8, column 1:

• name: filter_name ^ File "/home/ubuntu/TmLibrary/src/tmlib/workflow/jterator/project.py", line 212, in _create_handles 'description': f.read() File "/home/ubuntu/TmLibrary/src/tmlib/readers.py", line 182, in read return load_yaml(self._stream.read()) File "/home/ubuntu/TmLibrary/src/tmlib/readers.py", line 162, in load_yaml return ruamel.yaml.load(string, ruamel.yaml.RoundTripLoader) File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/main.py", line 75, in load return loader.get_single_data() File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/constructor.py", line 59, in get_single_data node = self.get_single_node() File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/composer.py", line 48, in get_single_node document = self.compose_document() File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/composer.py", line 68, in compose_document node = self.compose_node(None, None) File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/composer.py", line 99, in compose_node node = self.compose_mapping_node(anchor) File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/composer.py", line 151, in compose_mapping_node while not self.check_event(MappingEndEvent): File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/parser.py", line 116, in check_event self.current_event = self.state() File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/parser.py", line 505, in parse_block_mapping_key token.start_mark)

Copied from original issue: TissueMAPS/JtLibrary#9

[riccardomurri]

From @SaadiaIftikhar on June 2, 2016 13:10

Do you have any your pipeline, which I can try on my data to check - if my yaml file is not correct?

I checked online on yaml parser, the pipeline look ok.

[riccardomurri]

From @SaadiaIftikhar on June 2, 2016 13:13

---

description: '' input: channels:

• name: cycle-0-wavelength-1 correct: true pipeline: -active: true handles: handles/smooth_image_dapi.handles.yaml module: smooth_image.py

[riccardomurri]

From @SaadiaIftikhar on June 2, 2016 13:13

Output { "pipeline": null, "-active": true, "description": "", "input": { "channels": [ { "name": "cycle-0-wavelength-1", "correct": true } ] }, "handles": "handles/smooth_image_dapi.handles.yaml", "module": "smooth_image.py" }

[riccardomurri]

From @SaadiaIftikhar on June 2, 2016 13:17

and this one:

---

description: '' input: channels:

• name: cycle-0-wavelength-1 correct: true

pipeline:

• active: true handles: handles/smooth_image_dapi.handles.yaml module: smooth_image.py

[riccardomurri]

From @SaadiaIftikhar on June 2, 2016 13:17

Output { "input": { "channels": [ { "name": "cycle-0-wavelength-1", "correct": true } ] }, "pipeline": [ { "active": true, "handles": "handles/smooth_image_dapi.handles.yaml", "module": "smooth_image.py" } ], "description": "" }

[riccardomurri]

From @hackermd on June 2, 2016 13:34

The problem was in your smoth_image_dapi.handles.yaml file, which was not formatted properly. I fixed it.

[riccardomurri]

From @hackermd on June 2, 2016 13:39

I included the filename in the error message when reading of a file fails.

[riccardomurri]

From @SaadiaIftikhar on June 2, 2016 13:43

ubuntu@frontend001:/data/storage/experiments/experiment_1/workflow/jterator_segment_2d_cells_40x_20160602$ jterator -v 1 -p segment_2d_cells_40x_20160602 check

_ _                _

() | _ _ | | tmlib.workflow.jterator (0.0.7) | | / -) '/ ` | / \ '| Image analysis pipeline engine / |**|| **,|/_| https://github.com/TissueMAPS/TmLibrary |/

2016-06-02 13:43:18 | INFO | tmlib.workflow.jterator.cli | check pipe and handle descriptor files 2016-06-02 13:43:18 | INFO | tmlib.workflow.jterator.checkers | pipeline description check successful 2016-06-02 13:43:18 | INFO | tmlib.workflow.jterator.checkers | handles descriptions check successful

FAILED: The value of "key" of input item #0 with name "pipeline" in handles file "/data/storage/experiments/experiment_1/workflow/jterator_segment_2d_cells_40x_20160602/handles/smooth_image_dapi.handles.yaml" is not created upstream in the pipeline: "image" File "/home/ubuntu/TmLibrary/src/tmlib/workflow/cli.py", line 371, in main cli_instance(arguments) File "/home/ubuntu/TmLibrary/src/tmlib/workflow/cli.py", line 430, in call method(**method_args) File "/home/ubuntu/TmLibrary/src/tmlib/workflow/jterator/cli.py", line 48, in check self.api_instance.check_pipeline() File "/home/ubuntu/TmLibrary/src/tmlib/workflow/jterator/api.py", line 102, in check_pipeline checker.check_all() File "/home/ubuntu/TmLibrary/src/tmlib/workflow/jterator/checkers.py", line 365, in check_all self.check_pipeline_io() File "/home/ubuntu/TmLibrary/src/tmlib/workflow/jterator/checkers.py", line 346, in check_pipeline_io input_handle.key)

[riccardomurri]

From @SaadiaIftikhar on June 2, 2016 13:44

another error in the above comment ...

[riccardomurri]

From @hackermd on June 2, 2016 13:48

You need of course provide a valid input. "image" was not created upstream in the pipeline (as stated in the error message).

[riccardomurri]

From @SaadiaIftikhar on June 2, 2016 14:20

one log file:

FAILED: No row was found for one() File "/home/ubuntu/TmLibrary/src/tmlib/workflow/cli.py", line 371, in main cli_instance(arguments) File "/home/ubuntu/TmLibrary/src/tmlib/workflow/cli.py", line 430, in call method(**method_args) File "/home/ubuntu/TmLibrary/src/tmlib/workflow/cli.py", line 486, in run api.run_job(batch) File "/home/ubuntu/TmLibrary/src/tmlib/workflow/jterator/api.py", line 435, in run_job tm.Channel.experiment_id == self.experiment_id File "/home/ubuntu/.local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2534, in one raise orm_exc.NoResultFound("No row was found for one()")

[riccardomurri]

From @hackermd on June 2, 2016 14:34

There is no illumination statistics file for this channel. See log output and increase verbosity -v if required.

[riccardomurri]

From @SaadiaIftikhar on June 2, 2016 15:29

     created_at         |         updated_at         | id |      name       | channel_id | cycle_id 

----------------------------+----------------------------+----+-----------------+------------+---------- 2016-05-04 14:48:51.198063 | 2016-05-04 14:48:51.198063 | 1 | illumstats_6.h5 | 6 | 2 2016-05-04 14:48:56.261824 | 2016-05-04 14:48:56.261824 | 2 | illumstats_5.h5 | 5 | 2 2016-05-04 14:49:02.89459 | 2016-05-04 14:49:02.89459 | 3 | illumstats_7.h5 | 7 | 2 2016-05-04 14:54:11.659848 | 2016-05-04 14:54:11.659848 | 4 | illumstats_8.h5 | 8 | 2 (4 rows)

tissuemaps=> select * from illumstats_files_id_seq; sequence_name | last_value | start_value | increment_by | max_value | min_value | cache_value | log_cnt | is_cycled | is_called -------------------------+------------+-------------+--------------+---------------------+-----------+-------------+---------+-----------+----------- illumstats_files_id_seq | 4 | 1 | 1 | 9223372036854775807 | 1 | 1 | 32 | f | t (1 row)

tissuemaps=> select * from channels; created_at | updated_at | id | name | index | wavelength | experiment_id ----------------------------+----------------------------+----+----------------------+-------+------------+--------------- 2016-05-04 13:38:02.21998 | 2016-05-04 13:38:02.21998 | 5 | cycle-0_wavelength-1 | 0 | 1 | 1 2016-05-04 13:38:02.235177 | 2016-05-04 13:38:02.235177 | 6 | cycle-0_wavelength-2 | 1 | 2 | 1 2016-05-04 13:38:03.5074 | 2016-05-04 13:38:03.5074 | 7 | cycle-0_wavelength-3 | 2 | 3 | 1 2016-05-04 13:38:04.797519 | 2016-05-04 13:38:04.797519 | 8 | cycle-0_wavelength-5 | 3 | 5 | 1 (4 rows)

[riccardomurri]

From @SaadiaIftikhar on June 2, 2016 15:30

There are 4 illuminstat files in the db ...

[riccardomurri]

From @hackermd on June 2, 2016 15:34

Please post the full error trace (including the command) so that I can recapitulate what you did.

[riccardomurri]

From @hackermd on June 2, 2016 15:38

I just tried

jterator -v 1 -p segment_2d_cells_40x_20160602 run -j 1

after fixing two things in your handles file the run completed successfully.

How did you get the error? I cannot reproduce.

[riccardomurri]

From @SaadiaIftikhar on June 2, 2016 16:14

ubuntu@frontend001:~$ jterator -v 1 -p segment_2d_cells_40x_20160602 init

_ _                _

() | _ _ | | tmlib.workflow.jterator (0.0.7) | | / -) '/ ` | / \ '| Image analysis pipeline engine / |**|| **,|/_| https://github.com/TissueMAPS/TmLibrary |/

2016-06-02 16:10:35 | INFO | tmlib.workflow.cli | delete previous job output 2016-06-02 16:10:35 | INFO | tmlib.workflow.jterator.api | delete existing mapobject types 2016-06-02 16:10:35 | INFO | tmlib.workflow.cli | create batches 2016-06-02 16:10:35 | INFO | tmlib.workflow.jterator.checkers | pipeline description check successful 2016-06-02 16:10:36 | INFO | tmlib.workflow.jterator.checkers | handles descriptions check successful 2016-06-02 16:10:36 | INFO | tmlib.workflow.jterator.checkers | pipeline IO check successful 2016-06-02 16:10:38 | INFO | tmlib.workflow.cli | write batches to files 2016-06-02 16:10:41 | INFO | tmlib | COMPLETED ubuntu@frontend001:~$ jterator -v 1 -p segment_2d_cells_40x_20160602 submit

_ _                _

() | _ _ | | tmlib.workflow.jterator (0.0.7) | | / -) '/ ` | / \ '| Image analysis pipeline engine / |**|| **,|/_| https://github.com/TissueMAPS/TmLibrary |/

2016-06-02 16:10:49 | INFO | tmlib.workflow.cli | create all jobs 2016-06-02 16:10:50 | INFO | tmlib.workflow.api | create run jobs for submission 9 2016-06-02 16:10:50 | INFO | tmlib.workflow.api | create collect job for submission 9 2016-06-02 16:10:50 | INFO | tmlib.workflow.utils | create GC3Pie store using "tasks" SQL table 2016-06-02 16:10:53 | INFO | tmlib.workflow.utils | create GC3Pie engine 2016-06-02 16:10:53 | INFO | tmlib.workflow.cli | submit and monitor jobs 2016-06-02 16:10:56 | INFO | tmlib.workflow.cli | elapsed time: 0:00:03.003114 2016-06-02 16:10:56 | INFO | tmlib.workflow.cli | progress... +------------------+------------------------+-----------+----------+----------+-----------------+-------------+---------------------+-------+ | Name | Type | State | Done (%) | ExitCode | Time (HH:MM:SS) | Memory (MB) | CPU Time (HH:MM:SS) | ID | +------------------+------------------------+-----------+----------+----------+-----------------+-------------+---------------------+-------+ | jterator | WorkflowStep | NEW | 0.00 | | | | | 14536 | | jterator_run | SingleRunJobCollection | SUBMITTED | 0.00 | | | | | 14537 | | jterator_collect | CollectJob | NEW | 0.00 | | | | | 14730 | +------------------+------------------------+-----------+----------+----------+-----------------+-------------+---------------------+-------+ 2016-06-02 16:11:11 | INFO | tmlib.workflow.cli | elapsed time: 0:00:17.519243 2016-06-02 16:11:11 | INFO | tmlib.workflow.cli | progress... +------------------+------------------------+-----------+----------+----------+-----------------+-------------+---------------------+-------+ | Name | Type | State | Done (%) | ExitCode | Time (HH:MM:SS) | Memory (MB) | CPU Time (HH:MM:SS) | ID | +------------------+------------------------+-----------+----------+----------+-----------------+-------------+---------------------+-------+ | jterator | WorkflowStep | SUBMITTED | 0.00 | | | | | 14536 | | jterator_run | SingleRunJobCollection | RUNNING | 0.00 | | | | | 14537 | | jterator_collect | CollectJob | NEW | 0.00 | | | | | 14730 | +------------------+------------------------+-----------+----------+----------+-----------------+-------------+---------------------+-------+ 2016-06-02 16:11:18 | INFO | tmlib.workflow.cli | elapsed time: 0:00:25.105640 2016-06-02 16:11:18 | INFO | tmlib.workflow.cli | progress... +------------------+------------------------+---------+----------+----------+-----------------+-------------+---------------------+-------+ | Name | Type | State | Done (%) | ExitCode | Time (HH:MM:SS) | Memory (MB) | CPU Time (HH:MM:SS) | ID | +------------------+------------------------+---------+----------+----------+-----------------+-------------+---------------------+-------+ | jterator | WorkflowStep | RUNNING | 0.00 | | | | | 14536 | | jterator_run | SingleRunJobCollection | RUNNING | 0.00 | | | | | 14537 | | jterator_collect | CollectJob | NEW | 0.00 | | | | | 14730 | +------------------+------------------------+---------+----------+----------+-----------------+-------------+---------------------+-------+ 2016-06-02 16:11:26 | INFO | tmlib.workflow.cli | elapsed time: 0:00:32.916183 2016-06-02 16:11:26 | INFO | tmlib.workflow.cli | progress... +------------------+------------------------+---------+----------+----------+-----------------+-------------+---------------------+-------+ | Name | Type | State | Done (%) | ExitCode | Time (HH:MM:SS) | Memory (MB) | CPU Time (HH:MM:SS) | ID | +------------------+------------------------+---------+----------+----------+-----------------+-------------+---------------------+-------+ | jterator | WorkflowStep | RUNNING | 0.00 | | | | | 14536 | | jterator_run | SingleRunJobCollection | RUNNING | 8.33 | | | | | 14537 | | jterator_collect | CollectJob | NEW | 0.00 | | | | | 14730 | +------------------+------------------------+---------+----------+----------+-----------------+-------------+---------------------+-------+ 2016-06-02 16:11:37 | INFO | tmlib.workflow.cli | elapsed time: 0:00:44.071537 2016-06-02 16:11:37 | INFO | tmlib.workflow.cli | progress... +------------------+------------------------+---------+----------+----------+-----------------+-------------+---------------------+-------+ | Name | Type | State | Done (%) | ExitCode | Time (HH:MM:SS) | Memory (MB) | CPU Time (HH:MM:SS) | ID | +------------------+------------------------+---------+----------+----------+-----------------+-------------+---------------------+-------+ | jterator | WorkflowStep | RUNNING | 0.00 | | | | | 14536 | | jterator_run | SingleRunJobCollection | RUNNING | 33.85 | | | | | 14537 | | jterator_collect | CollectJob | NEW | 0.00 | | | | | 14730 | +------------------+------------------------+---------+----------+----------+-----------------+-------------+---------------------+-------+ 2016-06-02 16:11:56 | INFO | tmlib.workflow.cli | elapsed time: 0:01:03.029933 2016-06-02 16:11:56 | INFO | tmlib.workflow.cli | progress... +------------------+------------------------+---------+----------+----------+-----------------+-------------+---------------------+-------+ | Name | Type | State | Done (%) | ExitCode | Time (HH:MM:SS) | Memory (MB) | CPU Time (HH:MM:SS) | ID | +------------------+------------------------+---------+----------+----------+-----------------+-------------+---------------------+-------+ | jterator | WorkflowStep | RUNNING | 0.00 | | | | | 14536 | | jterator_run | SingleRunJobCollection | RUNNING | 52.08 | | | | | 14537 | | jterator_collect | CollectJob | NEW | 0.00 | | | | | 14730 | +------------------+------------------------+---------+----------+----------+-----------------+-------------+---------------------+-------+ 2016-06-02 16:12:12 | INFO | tmlib.workflow.cli | elapsed time: 0:01:19.131718 2016-06-02 16:12:12 | INFO | tmlib.workflow.cli | progress... +------------------+------------------------+---------+----------+----------+-----------------+-------------+---------------------+-------+ | Name | Type | State | Done (%) | ExitCode | Time (HH:MM:SS) | Memory (MB) | CPU Time (HH:MM:SS) | ID | +------------------+------------------------+---------+----------+----------+-----------------+-------------+---------------------+-------+ | jterator | WorkflowStep | RUNNING | 0.00 | | | | | 14536 | | jterator_run | SingleRunJobCollection | RUNNING | 75.00 | | | | | 14537 | | jterator_collect | CollectJob | NEW | 0.00 | | | | | 14730 | +------------------+------------------------+---------+----------+----------+-----------------+-------------+---------------------+-------+ 2016-06-02 16:12:30 | INFO | tmlib.workflow.cli | elapsed time: 0:01:36.509209 2016-06-02 16:12:30 | INFO | tmlib.workflow.cli | progress... +------------------+------------------------+---------+----------+----------+-----------------+-------------+---------------------+-------+ | Name | Type | State | Done (%) | ExitCode | Time (HH:MM:SS) | Memory (MB) | CPU Time (HH:MM:SS) | ID | +------------------+------------------------+---------+----------+----------+-----------------+-------------+---------------------+-------+ | jterator | WorkflowStep | RUNNING | 0.00 | | | | | 14536 | | jterator_run | SingleRunJobCollection | RUNNING | 95.83 | | | | | 14537 | | jterator_collect | CollectJob | NEW | 0.00 | | | | | 14730 | +------------------+------------------------+---------+----------+----------+-----------------+-------------+---------------------+-------+ 2016-06-02 16:12:43 | INFO | tmlib.workflow.cli | elapsed time: 0:01:50.183163 2016-06-02 16:12:43 | INFO | tmlib.workflow.cli | progress... +------------------+------------------------+------------+----------+----------+-----------------+-------------+---------------------+-------+ | Name | Type | State | Done (%) | ExitCode | Time (HH:MM:SS) | Memory (MB) | CPU Time (HH:MM:SS) | ID | +------------------+------------------------+------------+----------+----------+-----------------+-------------+---------------------+-------+ | jterator | WorkflowStep | RUNNING | 50.00 | | | | | 14536 | | jterator_run | SingleRunJobCollection | TERMINATED | 100.00 | 0 | 1:30:54 | | 1:08:06.999000 | 14537 | | jterator_collect | CollectJob | NEW | 0.00 | | | | | 14730 | +------------------+------------------------+------------+----------+----------+-----------------+-------------+---------------------+-------+ 2016-06-02 16:12:49 | INFO | tmlib.workflow.cli | elapsed time: 0:01:56.271377 2016-06-02 16:12:49 | INFO | tmlib.workflow.cli | progress... +------------------+------------------------+------------+----------+----------+-----------------+-------------+---------------------+-------+ | Name | Type | State | Done (%) | ExitCode | Time (HH:MM:SS) | Memory (MB) | CPU Time (HH:MM:SS) | ID | +------------------+------------------------+------------+----------+----------+-----------------+-------------+---------------------+-------+ | jterator | WorkflowStep | RUNNING | 50.00 | 0 | | | | 14536 | | jterator_run | SingleRunJobCollection | TERMINATED | 100.00 | 0 | 1:30:54 | | 1:08:06.999000 | 14537 | | jterator_collect | CollectJob | SUBMITTED | 0.00 | | | | | 14730 | +------------------+------------------------+------------+----------+----------+-----------------+-------------+---------------------+-------+ 2016-06-02 16:12:54 | INFO | tmlib.workflow.cli | elapsed time: 0:02:01.028624 2016-06-02 16:12:54 | INFO | tmlib.workflow.cli | progress... +------------------+------------------------+------------+----------+----------+-----------------+-------------+---------------------+-------+ | Name | Type | State | Done (%) | ExitCode | Time (HH:MM:SS) | Memory (MB) | CPU Time (HH:MM:SS) | ID | +------------------+------------------------+------------+----------+----------+-----------------+-------------+---------------------+-------+ | jterator | WorkflowStep | RUNNING | 100.00 | 0 | | | | 14536 | | jterator_run | SingleRunJobCollection | TERMINATED | 100.00 | 0 | 1:30:54 | | 1:08:06.999000 | 14537 | | jterator_collect | CollectJob | TERMINATED | 100.00 | 1 | 0:00:04 | 1129 | 0:00:04 | 14730 | +------------------+------------------------+------------+----------+----------+-----------------+-------------+---------------------+-------+ 2016-06-02 16:12:59 | INFO | tmlib.workflow.cli | elapsed time: 0:02:05.787921 2016-06-02 16:12:59 | INFO | tmlib.workflow.cli | progress... +------------------+------------------------+------------+----------+----------+-----------------+-------------+---------------------+-------+ | Name | Type | State | Done (%) | ExitCode | Time (HH:MM:SS) | Memory (MB) | CPU Time (HH:MM:SS) | ID | +------------------+------------------------+------------+----------+----------+-----------------+-------------+---------------------+-------+ | jterator | WorkflowStep | TERMINATED | 100.00 | 1 | 1:30:58 | | 1:08:10.999000 | 14536 | | jterator_run | SingleRunJobCollection | TERMINATED | 100.00 | 0 | 1:30:54 | | 1:08:06.999000 | 14537 | | jterator_collect | CollectJob | TERMINATED | 100.00 | 1 | 0:00:04 | 1129 | 0:00:04 | 14730 | +------------------+------------------------+------------+----------+----------+-----------------+-------------+---------------------+-------+ 2016-06-02 16:13:03 | INFO | tmlib.workflow.cli | elapsed time: 0:02:10.056110 2016-06-02 16:13:03 | INFO | tmlib.workflow.cli | progress... +------------------+------------------------+------------+----------+----------+-----------------+-------------+---------------------+-------+ | Name | Type | State | Done (%) | ExitCode | Time (HH:MM:SS) | Memory (MB) | CPU Time (HH:MM:SS) | ID | +------------------+------------------------+------------+----------+----------+-----------------+-------------+---------------------+-------+ | jterator | WorkflowStep | TERMINATED | 100.00 | 1 | 1:30:58 | | 1:08:10.999000 | 14536 | | jterator_run | SingleRunJobCollection | TERMINATED | 100.00 | 0 | 1:30:54 | | 1:08:06.999000 | 14537 | | jterator_collect | CollectJob | TERMINATED | 100.00 | 1 | 0:00:04 | 1129 | 0:00:04 | 14730 | +------------------+------------------------+------------+----------+----------+-----------------+-------------+---------------------+-------+ 2016-06-02 16:13:06 | ERROR | tmlib.workflow.cli | jterator (id: 14536) failed with exitcode 1 2016-06-02 16:13:06 | ERROR | tmlib.workflow.cli | jterator_collect (id: 14730) failed with exitcode 1 2016-06-02 16:13:06 | INFO | tmlib | COMPLETED

[riccardomurri]

From @hackermd on June 2, 2016 18:16

The schema of some database tables was not up-to-date with the TmLibrary code. I altered the table structure accordingly.

[riccardomurri]

From @SaadiaIftikhar on June 3, 2016 9:50

I tried another pipeline with some more modules, its working fine. But for the data of Liberali lab needs 3D labelling method to proceed further on making pipelines.

[riccardomurri]

From @SaadiaIftikhar on June 10, 2016 9:8

Again the same error of illumination stats not found:

[riccardomurri]

From @SaadiaIftikhar on June 10, 2016 9:8

ubuntu@frontend001:/data/storage/experiments/experiment_1/workflow/jterator_segment_2d_cells_40x_20160602/handles$ jterator -v 1 -p segment_2d_cells_40x_20160602 run -j 100

_ _                _

() | _ _ | | tmlib.workflow.jterator (0.0.7) | | / -) '/ ` | / \ '| Image analysis pipeline engine / |**|| **,|/_| https://github.com/TissueMAPS/TmLibrary |/

2016-06-10 09:07:47 | INFO | tmlib.workflow.cli | read job description from batch file 2016-06-10 09:07:47 | INFO | tmlib.workflow.cli | run job #100 2016-06-10 09:07:47 | INFO | tmlib.workflow.jterator.api | handle pipeline input 2016-06-10 09:07:47 | WARNING | tmlib.workflow.jterator.project | no pipeline description provided in "/data/storage/experiments/experiment_1/workflow/jterator/segment_2d_cells_40x_20160602/segment_2d_cells_40x_20160602.pipe.yaml" 2016-06-10 09:07:47 | WARNING | tmlib.workflow.jterator.project | No handles files found: /data/storage/experiments/experiment_1/workflow/jterator/segment_2d_cells_40x_20160602/handles 2016-06-10 09:07:47 | INFO | tmlib.workflow.jterator.checkers | pipeline description check successful 2016-06-10 09:07:47 | INFO | tmlib.workflow.jterator.checkers | handles descriptions check successful 2016-06-10 09:07:47 | INFO | tmlib.workflow.jterator.checkers | pipeline IO check successful 2016-06-10 09:07:47 | INFO | tmlib.workflow.jterator.api | load images for channel "" 2016-06-10 09:07:47 | INFO | tmlib.workflow.jterator.api | load illumination statistics

FAILED: No illumination statistics file found for channel "" File "/home/ubuntu/TmLibrary/src/tmlib/workflow/cli.py", line 244, in main cli_instance(arguments) File "/home/ubuntu/TmLibrary/src/tmlib/workflow/cli.py", line 303, in call method(**method_args) File "/home/ubuntu/TmLibrary/src/tmlib/workflow/cli.py", line 359, in run api.run_job(batch) File "/home/ubuntu/TmLibrary/src/tmlib/workflow/jterator/api.py", line 433, in run_job 'channel "%s"' % channel_name

[riccardomurri]

From @SaadiaIftikhar on June 10, 2016 9:9

Database shows:

tissuemaps=> select * from channels; created_at | updated_at | id | name | index | wavelength | experiment_id ----------------------------+----------------------------+----+----------------------+-------+------------+--------------- 2016-05-04 13:38:02.21998 | 2016-05-04 13:38:02.21998 | 5 | cycle-0_wavelength-1 | 0 | 1 | 1 2016-05-04 13:38:02.235177 | 2016-05-04 13:38:02.235177 | 6 | cycle-0_wavelength-2 | 1 | 2 | 1 2016-05-04 13:38:03.5074 | 2016-05-04 13:38:03.5074 | 7 | cycle-0_wavelength-3 | 2 | 3 | 1 2016-05-04 13:38:04.797519 | 2016-05-04 13:38:04.797519 | 8 | cycle-0_wavelength-5 | 3 | 5 | 1 (4 rows)

tissuemaps=> select * from illumstats_files; created_at | updated_at | id | name | channel_id | cycle_id ----------------------------+----------------------------+----+-----------------+------------+---------- 2016-05-04 14:48:51.198063 | 2016-05-04 14:48:51.198063 | 1 | illumstats_6.h5 | 6 | 2 2016-05-04 14:48:56.261824 | 2016-05-04 14:48:56.261824 | 2 | illumstats_5.h5 | 5 | 2 2016-05-04 14:49:02.89459 | 2016-05-04 14:49:02.89459 | 3 | illumstats_7.h5 | 7 | 2 2016-05-04 14:54:11.659848 | 2016-05-04 14:54:11.659848 | 4 | illumstats_8.h5 | 8 | 2 (4 rows)

[riccardomurri]

From @hackermd on June 10, 2016 12:40

2016-06-10 09:07:47 | INFO | tmlib.workflow.jterator.api | load images for channel "" 2016-06-10 09:07:47 | INFO | tmlib.workflow.jterator.api | load illumination statistics

Apparently, you haven't specified a channel. Of course, it cannot find an illumination file for channel "".

[riccardomurri]

Closing since no-one on this thread is working on this project anymore.