angelolab / ark-analysis

Integrated pipeline for multiplexed image analysis
https://ark-analysis.readthedocs.io/en/latest/
MIT License
71 stars 25 forks source link

Deepcell upload loop cleaning #1023

Closed camisowers closed 1 year ago

camisowers commented 1 year ago

If you haven't already, please read through our contributing guidelines before opening your PR

What is the purpose of this PR?

Closes #628. Changes how we attempt to re-upload failed fov batches to the deepcell server. Also refactors create_deepcell_output() with cleanlier helper functions. The code will now also check for previously processed batches, to avoid re-zipping and re-uploading already segmented batches to deepcell.

How did you implement your changes

Currently, we attempt to re-upload a failed fov batches until success or 5 retry attempts are reached. A cleaner, more long term fix is to attempt to process each batch once, check if any batch .zip files are missing, and re-run the failed batches. The re-running continues until either a successful output or the function times out.

Remaining issues

camisowers commented 1 year ago

Jus a couple of small things.

For the Retry call at line 245 num_retries is not defined.

This PR was actually meant to scrap the retry a certain amount of times logic and instead implement recall run_deepcell_direct() in a loop until success or timeout. Looks like I missed some code that needed to be removed.