search

apple / ml-cvnets

CVNets: A library for training computer vision networks
https://apple.github.io/ml-cvnets
Other
1.76k stars 225 forks source link

Training process of Mask R-CNN crashes when workers parameter is greater than 0 #67

Open cywinski opened 1 year ago

cywinski commented 1 year ago

Hi,

I try to train the Mask R-CNN object detection model with EfficientNet B3 as backbone on my custom dataset. I train the model according to the configuration provided here on 2 GPUs with a batch size of 4 per every GPU. I start the training with a command presented here.

When I run the training with workers parameter greater than 0 after a couple of epochs/steps I get the following error, after which the training crushes:

Traceback (most recent call last):  File "/opt/venv/bin/cvnets-train", line 33, in <module>    sys.exit(load_entry_point('cvnets', 'console_scripts', 'cvnets-train')())
  File "/home/ir/pvc/ml-cvnets/main_train.py", line 234, in main_worker
    torch.multiprocessing.spawn(
  File "/opt/venv/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 240, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/opt/venv/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 198, in start_processes
    while not context.join():
  File "/opt/venv/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 140, in join
    raise ProcessExitedException(
torch.multiprocessing.spawn.ProcessExitedException: process 1 terminated with signal SIGKIL

/usr/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 28 leaked semaphore objects to clean up at shutdown

However, when I set the workers to 0, the training seems to work properly but very slowly - I was able to train the model for 23 epochs, where each epoch took over 3 hours.

Do you know what might be the cause of this problem?

tuobaye11 commented 1 year ago

how do i use ml-cvnet?
when i use follow command

export CFG_FILE="config/classification/imagenet/resnet.yaml" cvnets-train --common.config-file $CFG_FILE --common.results-loc classification_results

main_train: Command not found

farzadab commented 1 year ago

Original question: it seems to me like you’re running out of memory (RAM) and are getting OOM killed. I’d check the following a) RAM size (free) b) the dataset implementation c) monitor RAM during training.

Second question: you probably need to install the library with pip install -e .

On Fri, Apr 7, 2023 at 4:36 AM tuobaye11 @.***> wrote:

how do i use ml-cvnet? when i use follow command

export CFG_FILE="config/classification/imagenet/resnet.yaml" cvnets-train --common.config-file $CFG_FILE --common.results-loc classification_results

main_train: Command not found

— Reply to this email directly, view it on GitHub https://github.com/apple/ml-cvnets/issues/67#issuecomment-1500208153, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABGT6OWWQ6LEUTMMXJDE5R3W7733XANCNFSM6AAAAAAUO7VWVM . You are receiving this because you are subscribed to this thread.Message ID: @.***>