Parallelize preprocessing operations - Githubissues

KnottLab / codex

Pipeline processing CODEX output

MIT License

2 stars 1 forks source link

Parallelize preprocessing operations #14

Open nathanin opened 3 years ago

nathanin commented 3 years ago

Using ray to implement parallel processing of individual tiles where possible.

Parallelizable operations:

[x] EDOF (including reading tiles from disk)
[ ] cycle alignment calculation
[ ] stitching initialization

Needs testing + additional work:

[ ] making reference DAPI image and the current cycle / channel image available in shared memory via ray.put()
[ ] is parallel reading during edof actually faster in all hardware scenarios ?
[ ] what is the memory profile? can we estimate the required memory from the number of tiles ?
[ ] Singularity image with ray preinstalled, for use on clusters + run script.