1./0.merge-single-cells error

[ErinWeisbart]

ERROR:root:Uncaught exception
Traceback (most recent call last):
  File "recipe/1.generate-profiles/0.merge-single-cells.py", line 55, in <module>
    experiment_config=experiment_config_file,
  File "config/utils/config_utils.py", line 171, in process_configuration
    for x in sites
  File "config/utils/config_utils.py", line 174, in <listcomp>
    for compart_file in compartments
  File "config/utils/config_utils.py", line 174, in <genexpr>
    for compart_file in compartments
OSError: [Errno 5] Input/output error: '/home/ubuntu/bucket/projects/2018_11_20_Periscope_Calico/workspace/analysis/20200805_A549_WG_Screen/CP186F-Well1-12'

[ErinWeisbart]

Code around error location

    # Assert all compartment files are present in each site folder before populating
    compartments = [f"{x}.csv" for x in file_info["options"]["core"]["compartments"]]
    sites = [
        x
        for x in sites
        if all(
            compart_file in os.listdir(pathlib.Path(input_analysis_dir / x))
            for compart_file in compartments
        )
    ]

[ErinWeisbart]

file_info["options"]["core"]["compartments"] = Cells, Nuclei, Cytoplasm (default) Locally downloaded those .csvs and confirmed they were not archived or corrupted. So I/O error seems to be stochastic?

[ErinWeisbart]

Can run code on locally downloaded files and don't get error:

import pathlib
import os
input_analysis_dir = pathlib.Path('/Users/eweisbar/Desktop/20200805_A549_WG_Screen/')
ignore_files = ['.DS_Store']
sites = [x.name for x in input_analysis_dir.iterdir() if x.name not in ignore_files]
print (sites)
complist = ["Cells","Nuclei","Cytoplasm"]
compartments = [f"{x}.csv" for x in complist]
sites = [
    x
    for x in sites
    if all(
        compart_file in os.listdir(pathlib.Path(input_analysis_dir / x))
        for compart_file in compartments
    )
]
print (sites)

['CP186F-Well1-12', 'CP186A-Well1-26', 'CP186A-Well1-27'] ['CP186F-Well1-12', 'CP186A-Well1-26', 'CP186A-Well1-27']

[ErinWeisbart]

Closed by https://github.com/broadinstitute/pooled-cell-painting-profiling-template/pull/31 broadinstitute/pooled-cell-painting-profiling-recipe@e8f72fb broadinstitute/pooled-cell-painting-profiling-recipe@8afea28