AllenCell / napari-allencell-segmenter

A plugin that enables image segmentation provided by Allen Institute for Cell Science
BSD 3-Clause "New" or "Revised" License
16 stars 6 forks source link

Determine / document which systems we intend to support training / executing models on #154

Open hughes036 opened 1 year ago

hughes036 commented 1 year ago

We have already had quite a bit of trouble running example training on the systems that we have access to (our macbooks, Ubuntu workstations). As of now, here is what we have attempted, the result, and the blockers:

System Result Blocker Details Solution Related Issue
macOS Monterey 12.6 Intel i7 Fails C++ compile error (at runtime) fatal error: 'omp.h' file not found. AllenCellModeling/aics-im2im#184
macOS Ventura 13.3.1 Intel i7 Fails C++ compile error (at runtime) libomp.dylib not found. brew install libomp
export DYLD_LIBRARY_PATH=/usr/local/opt/libomp/lib:/usr/local/lib
macOS Monterey 12.4 Apple M1 Fails C++ compile error (at runtime) fatal error: 'omp.h' file not found. AllenCellModeling/aics-im2im#184
Ubuntu 16 Fails GPU driver runtime error RuntimeError: The NVIDIA driver on your system is too old (found version 9010). Please update your GPU driver. Update GPU driver from nvidia.com
OR install PyTorch version compiled with current CUDA driver.
Ubuntu 20 (EC2) Succeeds . . . .
Slurm (CPU) . . . . .
Slurm (GPU) . . . . .
AWS cluster (GPU) . . . . .

In all cases, the setup steps were:

And the experiment run was python aics_im2im/train.py experiment=im2im/segmentation.yaml trainer=cpu

hughes036 commented 1 year ago

We also have problems running napari + our plugin on the EC2 instances which we are running cyto-dl on. So, we don't have a system that will run both.