search

cchen-cc / MA-SAM

PyTorch implementation for MA-SAM
Apache License 2.0
131 stars 6 forks source link

NAN appears at training #16

Open ZhangZ7112 opened 6 months ago

ZhangZ7112 commented 6 months ago

Excuse me. When I reproduce the work to train on the provided preprocessed data of BTCV dataset. I didn't make any other changes except 1)using a single GPU (RTX 4090) instead; 2) using pre-trained SAM ViT_B. However, NAN appears at about epoch 4/400.

I insert some code snippets to help find the reason, through which I find the origin of NAN occurs no later than the output['low_res_logits'] . 图片 图片

Could you please offer some hints to further identify and resolve the problem?

ZhangHaoyang493 commented 3 weeks ago

I have encounter the same problem. I have no idea on how to solve it.

lihe-CV commented 5 days ago

I have encounter the same problem, can you solve it?