search

PNNL-CompBio / coderdata

Automation scripts and benchmark dataset package for cancer drug prediction deep learning models.
Other
11 stars 3 forks source link

Add more robustness in Synapse file download scripts. #239

Closed jjacobson95 closed 1 week ago

jjacobson95 commented 3 weeks ago

Both BeatAML and MPNST fail randomly (and somewhat often) due to synapse not always downloading all of the files. This causes a lot of reruns when attempting to check the that the entire build process is working and causes a loss of time. As such, I think this should be worked on soon because it will make future debugging faster as well.

In my most recent case, BeatAML failed due to a missing file and then I re-built from scratch and MPNST failed due to a missing file and now I am running this again.

We should add checks that all files are present and if any are missing, we could attempt to download them X more times instead of exiting.

jjacobson95 commented 3 weeks ago

This issue may be less prevalent than I thought. Turns out ensembl removed their "current" directory from this repo and it was causing beataml to fail.