Compressed all adata objects except for the initial one due to errors
Removed some commented-out code
Added revision arg to scvi docker image name
Using CUDA version 12.3
Fixed compatibility issues with NVIDIA driver, CUDA, and JAX
Parallelized scanpy
--
The tasks that use GPU in cluster data sub wf: integrate_sample_data and annotate_cells will give this message:
/usr/local/lib/python3.10/site-packages/lightning/pytorch/trainer/connectors/data_connector.py:441: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=15` in the `DataLoader` to improve performance.
However, when I set scvi.settings.num_threads = 15, it gives this message:
/opt/Python-3.10.12/Lib/multiprocessing/popen_fork.py:66: RuntimeWarning: os.fork() was called. os.fork() is incompatible with multithreaded code, and JAX is multithreaded, so this will likely lead to a deadlock.
I think what's happening is that JAX is using GPU and PyTorch is detecting the CPUs available
-- The tasks that use GPU in cluster data sub wf:
integrate_sample_dataandannotate_cellswill give this message:However, when I set
scvi.settings.num_threads = 15, it gives this message:I think what's happening is that JAX is using GPU and PyTorch is detecting the CPUs available