search

deeplearning-wisc / vos

source code for ICLR'22 paper "VOS: Learning What You Don’t Know by Virtual Outlier Synthesis"
Apache License 2.0
310 stars 54 forks source link

Find it stucks in 12000 iter when trained vos on VOC #36

Closed Zigars closed 1 year ago

Zigars commented 2 years ago

Using main-MLP branch, and training command：

python train_net_gmm.py --dataset-dir /data/haitaozhang/dataset/object_detection/VOC --num-gpus 4 --config-file VOC-Detection/faster-rcnn/vos.yaml --random-seed 0 --resume

image

it's stucking in 12000 iter, and gpus usage do not work. image

how can I solve this problem, maybe need use the main branch?

Zigars commented 2 years ago

I switch to main branch, and this problem review again, also I change the backbone to regnetx, it doesn't work either, train step was still stucking in 12000 iter, which the default starting vos iter.

d12306 commented 2 years ago

hi, @Zigars , thanks for the interest in our codebase. not sure about your machine, i think the speed of vos will be slower than the vanilla training, but it will not stuck.

could you double check whether it is caused by the slower training speed?

d12306 commented 1 year ago

@Zigars , hi! do you figure out this issue? i actually encountered the same question when I try to reproduce my results. is it related to the version of detectron2 and torch, or cuda version?

d12306 commented 1 year ago

never mind, i found that we need to use torch 1.7.1 will work.