When I train my own dataset(My dataset has six classes, and I have divided it into five base classes and a new class. I have replaced the number and names of classes in the code with my own dataset's class names and numbers) use the configs/detection/meta_rcnn/voc/split1/meta-rcnn_r101_c4_8xb4_voc-split1_base-training.py,
I only have a 3090 GPU, and my settings are as follows
configs/detection/base/datasets/nway_kshot/base_voc.py
data = dict(
samples_per_gpu=4,
workers_per_gpu=1,
model_init=dict(
samples_per_gpu=8,
workers_per_gpu=1,
configs/detection/meta_rcnn/voc/split1/meta-rcnn_r101_c4_8xb4_voc-split1_base-training.py
evaluation = dict(interval=6000)
lr_config = dict(warmup_iters=300, step=[1600])
optimizer = dict(lr=0.00001)
During the basic training phase, when iterating 950-1000 times, loss will suddenly become NAN
Please help me, I will be very grateful to you!!!
When I train my own dataset(My dataset has six classes, and I have divided it into five base classes and a new class. I have replaced the number and names of classes in the code with my own dataset's class names and numbers) use the configs/detection/meta_rcnn/voc/split1/meta-rcnn_r101_c4_8xb4_voc-split1_base-training.py, I only have a 3090 GPU, and my settings are as follows configs/detection/base/datasets/nway_kshot/base_voc.py data = dict( samples_per_gpu=4, workers_per_gpu=1, model_init=dict( samples_per_gpu=8, workers_per_gpu=1, configs/detection/meta_rcnn/voc/split1/meta-rcnn_r101_c4_8xb4_voc-split1_base-training.py evaluation = dict(interval=6000) lr_config = dict(warmup_iters=300, step=[1600]) optimizer = dict(lr=0.00001) During the basic training phase, when iterating 950-1000 times, loss will suddenly become NAN
Please help me, I will be very grateful to you!!!