search

DataBiosphere / toil

A scalable, efficient, cross-platform (Linux/macOS) and easy-to-use workflow engine in pure Python.
http://toil.ucsc-cgl.org/.
Apache License 2.0
900 stars 240 forks source link

toil clean does not delete job store buckets #754

Closed dyollluap closed 8 years ago

dyollluap commented 8 years ago

I have 2 jobstore buckets that remain persistent after using toil clean, more recent buckets (created in the past week) were cleaned out fine as normal. The message suggests they may have already been deleted, though still present on the s3 dashboard.

mesosbox@ip-172-31-6-148:~$ toil clean aws:us-west-2:plloyd-20160309 INFO:toil.lib.bioio:Logging set at level: INFO INFO:toil.utils.toilClean:Parsed arguments INFO:toil.utils.toilClean:Checking if we have files for toil INFO:toil.utils.toilClean:The specified JobStore does not exist, it may have already been deleted

mesosbox@ip-172-31-6-148:~$ toil clean aws:us-west-2:plloyd-20160318 INFO:toil.lib.bioio:Logging set at level: INFO INFO:toil.utils.toilClean:Parsed arguments INFO:toil.utils.toilClean:Checking if we have files for toil INFO:toil.utils.toilClean:The specified JobStore does not exist, it may have already been deleted

cket commented 8 years ago

Could this be because of s3's eventual consistency? You might want to try waiting a while and check in the buckets again later and see if they still show up. It's possible they really are deleted but the console doesn't reflect that fact yet.

hannes-ucsc commented 8 years ago

If what @cket suggests doesn't help, you may have attempted the clean earlier and it failed or you may have run the pipeline with --clean=onsuccess (or without --clean since onsuccess is the default) or the workflow may have failed in some weird way. Keep an eye on this, because the interesting bit would be the stacktrace of the first failed clean or the failed workflow.

hannes-ucsc commented 8 years ago

@dyollluap, please provide info and reopen if necessary.