search

InstituteforDiseaseModeling / malaria-model_validation

1 stars 2 forks source link

Analyzer workitems failing (ndiop_1993 and chonyi_1999) #63

Open krosenfeld-IDM opened 1 year ago

krosenfeld-IDM commented 1 year ago

I installed the package and ran the simulations using the snakemake workflow

pip install -e . -i https://packages.idmod.org/api/pypi/pypi-production/simple
cd simulations
snakemake -j

The simulations seem to have run successfully, but two analyzers, ndiop_1993 and chonyi_1999 are failing with the similar error messages:

Error message:
Errors encountered during file download, directory creation, and/or symlink processing. The most common cause of this error is the use of long file paths and/or names. See logs for more details.
CorrelationId: 08dd7c56-9ad5-4a03-9e3a-13dade66b63e

I am not sure which log would contain more details. There is no output associated with the workitem. My COMPS_log.log contains:

07/20 09:29:52 [DEBUG] [140120075540288] [COMPS.Data.SerializableEntity] OutputFileMetadata --> {'Id': '_internal_id', 'Length': 'length', 'FriendlyName': 'friendly_name', 'PathFromRoot': 'path_from_root', 'Url': 'url', 'MimeType': 'mime_type'}
07/20 09:29:52 [DEBUG] [140120075540288] [COMPS.Data.SerializableEntity] AssetFile --> {'FileName': 'file_name', 'MD5Checksum': 'md5_checksum', 'Length': 'length', 'Uri': 'uri'}
07/20 09:29:52 [DEBUG] [140120075540288] [COMPS.Data.SerializableEntity] AssetCollectionFile --> {'RelativePath': 'relative_path', 'Tags': 'tags', 'FileName': 'file_name', 'MD5Checksum': 'md5_checksum', 'Length': 'length', 'Uri': 'uri'}
07/20 09:29:52 [DEBUG] [140120075540288] [COMPS.Data.SerializableEntity] AssetCollection --> {'Id': 'id', 'DateCreated': 'date_created', 'Tags': 'tags', 'Assets': 'assets'}
07/20 09:29:52 [DEBUG] [140120075540288] [COMPS.Data.SerializableEntity] Configuration --> {'EnvironmentName': 'environment_name', 'SimulationInputArgs': 'simulation_input_args', 'WorkingDirectoryRoot': 'working_directory_root', 'ExecutablePath': 'executable_path', 'NodeGroupName': 'node_group_name', 'MaximumNumberOfRetries': 'maximum_number_of_retries', 'Priority': 'priority', 'MinCores': 'min_cores', 'MaxCores': 'max_cores', 'Exclusive': 'exclusive', 'AssetCollectionId': 'asset_collection_id'}
07/20 09:29:52 [DEBUG] [140120075540288] [COMPS.Data.SerializableEntity] SimulationFile --> {'FileType': 'file_type', 'Description': 'description', 'FileName': 'file_name', 'MD5Checksum': 'md5_checksum', 'Length': 'length', 'Uri': 'uri'}
07/20 09:29:52 [DEBUG] [140120075540288] [COMPS.Data.SerializableEntity] WorkItemFile --> {'FileType': 'file_type', 'Description': 'description', 'FileName': 'file_name', 'MD5Checksum': 'md5_checksum', 'Length': 'length', 'Uri': 'uri'}
07/20 09:29:52 [DEBUG] [140120075540288] [COMPS.Data.SerializableEntity] HpcJob --> {'Id': '_internal_id', 'JobId': 'job_id', 'JobState': 'job_state', 'Priority': 'priority', 'WorkingDirectory': 'working_directory', 'OutputDirectorySize': 'output_directory_size', 'SubmitTime': 'submit_time', 'StartTime': 'start_time', 'EndTime': 'end_time', 'ErrorMessage': 'error_message', 'Configuration': 'configuration'}
07/20 09:29:52 [DEBUG] [140120075540288] [COMPS.Data.SerializableEntity] Simulation --> {'Id': 'id', 'ExperimentId': 'experiment_id', 'Name': 'name', 'Description': 'description', 'Owner': 'owner', 'DateCreated': 'date_created', 'LastModified': 'last_modified', 'SimulationState': 'state', 'ErrorMessage': 'error_message', 'Tags': 'tags', 'Configuration': 'configuration', 'Files': 'files', 'HPCJobs': 'hpc_jobs'}
07/20 09:29:52 [DEBUG] [140120075540288] [COMPS.Data.SerializableEntity] Experiment --> {'Id': 'id', 'SuiteId': 'suite_id', 'Name': 'name', 'Description': 'description', 'Owner': 'owner', 'DateCreated': 'date_created', 'LastModified': 'last_modified', 'Tags': 'tags', 'Configuration': 'configuration'}
07/20 09:29:52 [DEBUG] [140120075540288] [COMPS.Data.SerializableEntity] Suite --> {'Id': 'id', 'Name': 'name', 'Description': 'description', 'Owner': 'owner', 'DateCreated': 'date_created', 'LastModified': 'last_modified', 'Tags': 'tags', 'Configuration': 'configuration'}
07/20 09:29:52 [DEBUG] [140120075540288] [COMPS.Data.SerializableEntity] WorkItem --> {'Id': 'id', 'Name': 'name', 'Worker': 'worker', 'EnvironmentName': 'environment_name', 'Description': 'description', 'Owner': 'owner', 'DateCreated': 'date_created', 'LastModified': 'last_modified', 'State': 'state', 'ErrorMessage': 'error_message', 'HostName': 'host_name', 'WorkerInstanceId': 'worker_instance_id', 'Priority': 'priority', 'WorkingDirectory': 'working_directory', 'WorkingDirectorySize': 'working_directory_size', 'AssetCollectionId': 'asset_collection_id', 'Tags': 'tags', 'Files': 'files', 'Plugins': 'plugins'}
07/20 09:29:53 [DEBUG] [140120075540288] [COMPS.Client] Parsed login host: ('https', 'comps.idmod.org', None)
07/20 09:29:53 [DEBUG] [140120075540288] [COMPS.Client] REQUEST -> GET https://comps.idmod.org/api/Experiments/35911bda-5526-ee11-aa08-b88303911bc1 {'fields': 'Id,Name,SuiteId', 'children': 'Tags,Configuration'}
07/20 09:29:53 [DEBUG] [140120075540288] [COMPS.Client] REQUEST -> GET https://comps.idmod.org/api/Suites/b805d021-5626-ee11-aa08-b88303911bc1 {'fields': 'Id,Name', 'children': 'Tags,Configuration'}
07/20 09:29:53 [DEBUG] [140120075540288] [COMPS.Client] REQUEST -> GET https://comps.idmod.org/api/Experiments {'filters': 'SuiteId=b805d021-5626-ee11-aa08-b88303911bc1'}
07/20 09:29:53 [DEBUG] [140120075540288] [COMPS.Client] REQUEST -> GET https://comps.idmod.org/api/Simulations {'fields': 'Id,Name,ExperimentId,SimulationState', 'children': 'Tags,Files,Configuration', 'filters': 'ExperimentId=26f723cc-5526-ee11-aa08-b88303911bc1', 'count': 100000}
07/20 09:29:53 [DEBUG] [140120075540288] [COMPS.Client] REQUEST -> GET https://comps.idmod.org/api/Experiments/26f723cc-5526-ee11-aa08-b88303911bc1 {'children': 'Configuration'}
07/20 09:29:53 [DEBUG] [140120075540288] [COMPS.Client] REQUEST -> GET https://comps.idmod.org/api/Simulations {'fields': 'Id,Name,ExperimentId,SimulationState', 'children': 'Tags,Files,Configuration', 'filters': 'ExperimentId=27f723cc-5526-ee11-aa08-b88303911bc1', 'count': 100000}
07/20 09:29:53 [DEBUG] [140120075540288] [COMPS.Client] REQUEST -> GET https://comps.idmod.org/api/Experiments/27f723cc-5526-ee11-aa08-b88303911bc1 {'children': 'Configuration'}
07/20 09:29:53 [DEBUG] [140120075540288] [COMPS.Client] REQUEST -> GET https://comps.idmod.org/api/Simulations {'fields': 'Id,Name,ExperimentId,SimulationState', 'children': 'Tags,Files,Configuration', 'filters': 'ExperimentId=28f723cc-5526-ee11-aa08-b88303911bc1', 'count': 100000}
07/20 09:29:53 [DEBUG] [140120075540288] [COMPS.Client] REQUEST -> GET https://comps.idmod.org/api/Experiments/28f723cc-5526-ee11-aa08-b88303911bc1 {'children': 'Configuration'}
07/20 09:29:53 [DEBUG] [140120075540288] [COMPS.Client] REQUEST -> GET https://comps.idmod.org/api/Simulations {'fields': 'Id,Name,ExperimentId,SimulationState', 'children': 'Tags,Files,Configuration', 'filters': 'ExperimentId=29f723cc-5526-ee11-aa08-b88303911bc1', 'count': 100000}
07/20 09:29:53 [DEBUG] [140120075540288] [COMPS.Client] REQUEST -> GET https://comps.idmod.org/api/Experiments/29f723cc-5526-ee11-aa08-b88303911bc1 {'children': 'Configuration'}
07/20 09:29:53 [DEBUG] [140120075540288] [COMPS.Client] REQUEST -> GET https://comps.idmod.org/api/Simulations {'fields': 'Id,Name,ExperimentId,SimulationState', 'children': 'Tags,Files,Configuration', 'filters': 'ExperimentId=d38824d3-5526-ee11-aa08-b88303911bc1', 'count': 100000}
07/20 09:29:53 [DEBUG] [140120075540288] [COMPS.Client] REQUEST -> GET https://comps.idmod.org/api/Experiments/d38824d3-5526-ee11-aa08-b88303911bc1 {'children': 'Configuration'}

I am working on a RHEL8 machine with conda environment: 

# packages in environment at /home/krosenfeld/playpen/malaria-model_validation/env:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
appdirs                   1.4.4                    pypi_0    pypi
astor                     0.8.1                    pypi_0    pypi
astunparse                1.6.3                    pypi_0    pypi
attrs                     23.1.0                   pypi_0    pypi
backoff                   2.2.1                    pypi_0    pypi
beautifulsoup4            4.12.2                   pypi_0    pypi
bs4                       0.0.1                    pypi_0    pypi
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2023.5.7             hbcca054_0    conda-forge
certifi                   2023.5.7                 pypi_0    pypi
cffi                      1.15.1                   pypi_0    pypi
charset-normalizer        3.2.0                    pypi_0    pypi
click                     7.1.2                    pypi_0    pypi
coloredlogs               15.0.1                   pypi_0    pypi
configargparse            1.5.5                    pypi_0    pypi
connection-pool           0.0.3                    pypi_0    pypi
contourpy                 1.1.0                    pypi_0    pypi
cryptography              41.0.2                   pypi_0    pypi
cycler                    0.11.0                   pypi_0    pypi
datar                     0.12.2                   pypi_0    pypi
datrie                    0.8.2                    pypi_0    pypi
defusedxml                0.7.1                    pypi_0    pypi
diot                      0.2.2                    pypi_0    pypi
diskcache                 5.4.0                    pypi_0    pypi
docopt                    0.6.2                    pypi_0    pypi
docutils                  0.20.1                   pypi_0    pypi
dpath                     2.1.6                    pypi_0    pypi
emod-api                  1.29.7                   pypi_0    pypi
emod-malaria              0.0.58                   pypi_0    pypi
emodpy                    1.21.1                   pypi_0    pypi
emodpy-malaria            2.4.3                    pypi_0    pypi
executing                 1.2.0                    pypi_0    pypi
fastjsonschema            2.17.1                   pypi_0    pypi
filelock                  3.12.2                   pypi_0    pypi
fonttools                 4.41.0                   pypi_0    pypi
fpdf2                     2.7.4                    pypi_0    pypi
future                    0.18.3                   pypi_0    pypi
geographiclib             2.0                      pypi_0    pypi
gitdb                     4.0.10                   pypi_0    pypi
gitpython                 3.1.32                   pypi_0    pypi
humanfriendly             10.0                     pypi_0    pypi
idmtools                  1.7.8                    pypi_0    pypi
idmtools-models           1.7.8                    pypi_0    pypi
idmtools-platform-comps   1.7.8                    pypi_0    pypi
idna                      3.4                      pypi_0    pypi
importlib-metadata        6.8.0                    pypi_0    pypi
importlib-resources       6.0.0                    pypi_0    pypi
inflection                0.5.1                    pypi_0    pypi
jaraco-classes            3.3.0                    pypi_0    pypi
jeepney                   0.8.0                    pypi_0    pypi
jinja2                    3.1.2                    pypi_0    pypi
joblib                    1.3.1                    pypi_0    pypi
jsonschema                4.18.4                   pypi_0    pypi
jsonschema-specifications 2023.7.1                 pypi_0    pypi
jupyter-core              5.3.1                    pypi_0    pypi
keyring                   24.2.0                   pypi_0    pypi
kiwisolver                1.4.4                    pypi_0    pypi
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.1.0               he5830b7_0    conda-forge
libgomp                   13.1.0               he5830b7_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libsqlite                 3.42.0               h2797004_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
lz4                       4.3.2                    pypi_0    pypi
malaria-model-validation  1.2.0                    pypi_0    pypi
markupsafe                2.1.3                    pypi_0    pypi
matplotlib                3.7.2                    pypi_0    pypi
mizani                    0.9.2                    pypi_0    pypi
more-itertools            9.0.0                    pypi_0    pypi
natsort                   8.2.0                    pypi_0    pypi
nbformat                  5.9.1                    pypi_0    pypi
ncurses                   6.4                  hcb278e6_0    conda-forge
numpy                     1.25.1                   pypi_0    pypi
openssl                   3.1.1                hd590300_1    conda-forge
packaging                 21.3                     pypi_0    pypi
pandas                    2.0.3                    pypi_0    pypi
parse                     1.19.1                   pypi_0    pypi
patsy                     0.5.3                    pypi_0    pypi
pillow                    10.0.0                   pypi_0    pypi
pip                       23.2               pyhd8ed1ab_0    conda-forge
pipda                     0.12.0                   pypi_0    pypi
pipreqs                   0.4.13                   pypi_0    pypi
plac                      1.3.5                    pypi_0    pypi
platformdirs              3.9.1                    pypi_0    pypi
plotnine                  0.12.1                   pypi_0    pypi
pluggy                    1.0.0                    pypi_0    pypi
prodict                   0.8.18                   pypi_0    pypi
psutil                    5.9.5                    pypi_0    pypi
pulp                      2.7.0                    pypi_0    pypi
pycomps                   2.10.0                   pypi_0    pypi
pycparser                 2.21                     pypi_0    pypi
pygit2                    1.10.1                   pypi_0    pypi
pyparsing                 3.0.9                    pypi_0    pypi
pyproj                    3.6.0                    pypi_0    pypi
python                    3.9.16          h2782a2a_0_cpython    conda-forge
krosenfeld-IDM commented 1 year ago

Came back a few days later and tried rerunning:

snakemake -j

The chonyi_1999 work item and the ndiop_1993 work item ran this time. I didn't make any changes to files. I guess there is something buggy about how some temporary files or linking is going on?