636 enhance 3d image processing based on ma sam

lufre1 commented 1 week ago

added MA-SAM 3D adapters ideas. Implemented in specific 3D wrappers for the SAM encoder but also for the predictor

lufre1 commented 1 week ago

@constantinpape please have a look

constantinpape commented 1 week ago

This is now working more or less, with a few caveats.

The model does not take care of resizing the inputs. (Seems to be not implemented in MA-SAM either).
MA-SAM uses a patch shape of 512 x 512 and over-rides the build_sam functionality for this (it's probably easy to do this for us as well). See here.
Training is running, but throws a warning about non-leaf tensors (see below). So likely not everything is getting updated with gradients. Not sure why; MA-SAM is doing a similar job of initializing the model, see https://github.com/cchen-cc/MA-SAM/blob/main/MA-SAM/train.py#L77-L83.

UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being acce
ssed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on 
the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more infor
mations. (Triggered internally at /opt/conda/conda-bld/pytorch_1695392035891/work/build/aten/src/ATen/core/TensorBody.h:489.)

constantinpape commented 1 week ago

I did some minor refactoring and this should be good to go now. I ll merge it into dev.

computational-cell-analytics / micro-sam

636 enhance 3d image processing based on ma sam #639