Subprocess execution fails because the configuration file cannot be read

[ghadjigeorghiou]

Hello,

I was trying to automate the HistoQC script execution to include it in my workflow. I tried to use the subprocess python module to automatically execute the command through a terminal. I also changed the working directory so that everything would be executed from the HistoQC folder as recommended. However, once the qc-pipeline.py script is called, the execution exits with an error from the configparser indicating that the keyword pipeline could not be found in the config.ini file. I tested the command, being ran from the subprocess method, directly through the terminal and it works fine.

Since the error seems to be related with the subprocess module, I was wondering if there is anyway to import the qc-pipeline script in my code and change the main method to accept the output directories, input directories etc as parameters instead of using a terminal.

For example instead of running the command "python qc_pipeline.py -o /.TIA/wsi-processed -c config.ini -f -n 6 -s ./TIA/UserTemp/TCGA_22_04.svs " to be able to do something like:

import qc-pipeline qc-pipeline.main("./TIA/wsi-processed", config.ini, "f", 6, "s", "./TIA/UserTemp/TCGA_22.svs")

Error: 2020-03-02 01:48:30,577 - INFO - no display found. Using non-interactive Agg backend 2020-03-02 01:48:30,739 - INFO - Pipeline will use these steps: Traceback (most recent call last): File "/Users/ghadjigeorgiou/miniconda3/envs/TIA-36/lib/python3.6/configparser.py", line 1138, in _unify_values sectiondict = self._sections[section] KeyError: 'pipeline'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "qc_pipeline.py", line 183, in processQueue = load_pipeline(config) File "qc_pipeline.py", line 113, in load_pipeline steps = lconfig.get('pipeline', 'steps').splitlines() File "/Users/ghadjigeorgiou/miniconda3/envs/TIA-36/lib/python3.6/configparser.py", line 781, in get d = self._unify_values(section, vars) File "/Users/ghadjigeorgiou/miniconda3/envs/TIA-36/lib/python3.6/configparser.py", line 1141, in _unify_values raise NoSectionError(section) configparser.NoSectionError: No section: 'pipeline'

Thanks, George

[choosehappy]

seems the issue is that the config file isn't being loaded correctly, perhaps you need to use the full path?

On Mon, Mar 2, 2020 at 2:56 AM ghadjigeorghiou notifications@github.com wrote:

Hello,

I was trying to automate the HistoQC script execution to include it in my workflow. I tried to use the subprocess python module to automatically execute the command through a terminal. I also changed the working directory so that everything would be executed from the HistoQC folder as recommended. However, once the qc-pipeline.py script is called, the execution exits with an error from the configparser indicating that the keyword pipeline could not be found in the config.ini file. I tested the command, being ran from the subprocess method, directly through the terminal and it works fine.

Since the error seems to be related with the subprocess module, I was wondering if there is anyway to import the qc-pipeline script in my code and change the main method to accept the output directories, input directories etc as parameters instead of using a terminal.

For example instead of running the command "python qc_pipeline.py -o /.TIA/wsi-processed -c config.ini -f -n 6 -s ./TIA/UserTemp/TCGA_22_04.svs " to be able to do something like:

import qc-pipeline qc-pipeline.main("./TIA/wsi-processed", config.ini, "f", 6, "s", "./TIA/UserTemp/TCGA_22.svs")

Error: 2020-03-02 01:48:30,577 - INFO - no display found. Using non-interactive Agg backend 2020-03-02 01:48:30,739 - INFO - Pipeline will use these steps: Traceback (most recent call last): File "/Users/ghadjigeorgiou/miniconda3/envs/TIA-36/lib/python3.6/configparser.py", line 1138, in _unify_values sectiondict = self._sections[section] KeyError: 'pipeline'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "qc_pipeline.py", line 183, in processQueue = load_pipeline(config) File "qc_pipeline.py", line 113, in load_pipeline steps = lconfig.get('pipeline', 'steps').splitlines() File "/Users/ghadjigeorgiou/miniconda3/envs/TIA-36/lib/python3.6/configparser.py", line 781, in get d = self._unify_values(section, vars) File "/Users/ghadjigeorgiou/miniconda3/envs/TIA-36/lib/python3.6/configparser.py", line 1141, in _unify_values raise NoSectionError(section) configparser.NoSectionError: No section: 'pipeline'

Thanks, George

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/choosehappy/HistoQC/issues/155?email_source=notifications&email_token=ACJ3XTAQLXC3HZGTY2NE2MDRFMG5HA5CNFSM4K7KVAW2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IRRPW5A, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACJ3XTG6NANT6NN7ONSN2XTRFMG5HANCNFSM4K7KVAWQ .

[ghadjigeorghiou]

I have tried to use the full path but I get the same error.

Should I try to use HistoQC as a python package?

[choosehappy]

wel,l i would stick a breakpoint in the config loading part and see why its not loading properly:

https://github.com/choosehappy/HistoQC/blob/ca7c9edb6c025de9f4716dc5c3b5fb75d048a10c/qc_pipeline.py#L181

i suspect the error is exactly on this line:

https://github.com/choosehappy/HistoQC/blob/ca7c9edb6c025de9f4716dc5c3b5fb75d048a10c/qc_pipeline.py#L113

as it is the first one which attempts to use information from the configuration file

HistoQC was designed and tested to be a stand-alone application, but that said i see no reason it couldn't be encapsulated in something else

On Mon, Mar 2, 2020 at 2:25 PM ghadjigeorghiou notifications@github.com wrote:

I have tried to use the full path but I get the same error.

Should I try to use HistoQC as a python package?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/choosehappy/HistoQC/issues/155?email_source=notifications&email_token=ACJ3XTH4DLMG7URGRBOPFVTRFOXUDA5CNFSM4K7KVAW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENPJCZI#issuecomment-593400165, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACJ3XTBWCGWYYAJFYKGGLZTRFOXUDANCNFSM4K7KVAWQ .

[ghadjigeorghiou]

I managed to get it working by using the subprocess.call command instead of the subprocess.run command.

Example of code resulting in error: sub_out = subprocess.run(['python', histoqc_path, output_arg, main_config, '-f', threads_arg, '-s', path], stdout=PIPE, stderr=PIPE)

Working code: command = 'python ' + histoqc_path + ' ' + output_arg + ' ' + main_config + ' -f ' + threads_arg + ' -s ' + path sub_out2 = subprocess.call(command, shell=True)

[choosehappy]

cool, thanks for the update!

On Mon, Mar 2, 2020 at 2:47 PM ghadjigeorghiou notifications@github.com wrote:

Closed #155 https://github.com/choosehappy/HistoQC/issues/155.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/choosehappy/HistoQC/issues/155?email_source=notifications&email_token=ACJ3XTHEIUQUSMPSGTKD4PLRFO2HHA5CNFSM4K7KVAW2YY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOXAHUMYA#event-3088008800, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACJ3XTG7JUB4X3ZTBEVL5PDRFO2HHANCNFSM4K7KVAWQ .