MendelXu
commented
2 years ago Sorry for the late reply. I just spent a long vacation. What is your command to run the job?
Closed vefak closed 2 years ago
MendelXu
commented
2 years ago Sorry for the late reply. I just spent a long vacation. What is your command to run the job?
vefak
commented
2 years ago Sorry for the late reply. I just spent a long vacation. What is your command to run the job?
Hi thanks for replying. Here my command
python demo/image_demo.py planes.jpg configs/soft_teacher/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k.py work_dirs/iter_720000.pth --output work_dirs/
MendelXu
commented
2 years ago Are you executing it on WSL? Could you add some details like operating system, cuda version.
vefak
commented
2 years ago Are you executing it on WSL? Could you add some details like operating system, cuda version.
No I dont. Here my system information. And I am running on conda env
Distributor ID: Ubuntu
Description: Ubuntu 20.04.3 LTS
Release: 20.04
Codename: focalHere my conda env packages (Conda 4.10.3)
packages in environment at /home/vefak/Documents/anaconda3/envs/torch:
Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 4.5 1_gnu
addict 2.4.0 pypi_0 pypi
bzip2 1.0.8 h7f98852_4 conda-forge
ca-certificates 2021.10.26 h06a4308_2
certifi 2021.5.30 py36h5fab9bb_0 conda-forge
cffi 1.14.6 py36hc120d54_0 conda-forge
charset-normalizer 2.0.11 pypi_0 pypi
click 8.0.3 pypi_0 pypi
configparser 5.2.0 pypi_0 pypi
cpuonly 1.0 0 pytorch
cudatoolkit 11.3.1 h2bc3f7f_2
cudnn 8.2.1.32 h86fa8c9_0 conda-forge
cycler 0.11.0 pypi_0 pypi
dataclasses 0.8 pyh787bdff_2 conda-forge
docker-pycreds 0.4.0 pypi_0 pypi
ffmpeg 4.3 hf484d3e_0 pytorch
freetype 2.10.4 h0708190_1 conda-forge
future 0.18.2 py36h5fab9bb_3 conda-forge
gitdb 4.0.9 pypi_0 pypi
gitpython 3.1.18 pypi_0 pypi
gmp 6.2.1 h58526e2_0 conda-forge
gnutls 3.6.13 h85f3911_1 conda-forge
idna 3.3 pypi_0 pypi
importlib-metadata 4.8.3 pypi_0 pypi
install 1.3.5 pypi_0 pypi
intel-openmp 2021.4.0 h06a4308_3561
jpeg 9b h024ee3a_2
kiwisolver 1.3.1 pypi_0 pypi
lame 3.100 h7f98852_1001 conda-forge
ld_impl_linux-64 2.35.1 h7274673_9
libblas 3.9.0 12_linux64_mkl conda-forge
libcblas 3.9.0 12_linux64_mkl conda-forge
libffi 3.3 he6710b0_2
libgcc-ng 9.3.0 h5101ec6_17
libgomp 9.3.0 h5101ec6_17
libiconv 1.16 h516909a_0 conda-forge
liblapack 3.9.0 12_linux64_mkl conda-forge
libpng 1.6.37 h21135ba_2 conda-forge
libprotobuf 3.16.0 h780b84a_0 conda-forge
libstdcxx-ng 9.3.0 hd4cf53a_17
libtiff 4.0.9 he6b73bb_1 conda-forge
magma 2.5.4 h6103c52_2 conda-forge
matplotlib 3.3.4 pypi_0 pypi
mkl 2021.4.0 h06a4308_640
mmcv-full 1.3.9 dev_0 <develop>
mmdet 2.16.0 dev_0 <develop>
nccl 2.11.4.1 hdc17891_0 conda-forge
ncurses 6.3 h7f8727e_2
nettle 3.6 he412f7d_0 conda-forge
ninja 1.10.2 h4bd325d_0 conda-forge
numpy 1.19.5 py36hfc0c790_2 conda-forge
olefile 0.46 pyh9f0ad1d_1 conda-forge
opencv-python 4.5.5.62 pypi_0 pypi
openh264 2.1.1 h780b84a_0 conda-forge
openssl 1.1.1m h7f8727e_0
packaging 21.3 pypi_0 pypi
pathtools 0.1.2 pypi_0 pypi
pillow 8.4.0 pypi_0 pypi
pip 21.2.2 py36h06a4308_0
prettytable 2.5.0 pypi_0 pypi
promise 2.3 pypi_0 pypi
protobuf 3.19.4 pypi_0 pypi
psutil 5.9.0 pypi_0 pypi
pycocotools 2.0.4 pypi_0 pypi
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pyparsing 3.0.7 pypi_0 pypi
python 3.6.13 h12debd9_1
python-dateutil 2.8.2 pypi_0 pypi
python_abi 3.6 2_cp36m conda-forge
pytorch 1.9.0 cuda112py36h755b813_1 conda-forge
pytorch-gpu 1.9.0 cuda112py36h0bbbad9_1 conda-forge
pyyaml 6.0 pypi_0 pypi
readline 8.1.2 h7f8727e_1
requests 2.27.1 pypi_0 pypi
sentry-sdk 1.5.4 pypi_0 pypi
setuptools 58.0.4 py36h06a4308_0
shortuuid 1.0.8 pypi_0 pypi
six 1.16.0 pypi_0 pypi
sleef 3.5.1 h7f98852_1 conda-forge
smmap 5.0.0 pypi_0 pypi
sqlite 3.37.0 hc218d9a_0
ssod 0.0.1 dev_0 <develop>
subprocess32 3.5.4 pypi_0 pypi
termcolor 1.1.0 pypi_0 pypi
terminaltables 3.1.10 pypi_0 pypi
tk 8.6.11 h1ccaba5_0
torchaudio 0.9.0 py36 pytorch
torchvision 0.10.0 py36_cpu [cpuonly] pytorch
typing_extensions 4.0.1 pyha770c72_0 conda-forge
urllib3 1.26.8 pypi_0 pypi
wandb 0.10.31 pypi_0 pypi
wcwidth 0.2.5 pypi_0 pypi
wheel 0.37.1 pyhd3eb1b0_0
xz 5.2.5 h7b6447c_0
yapf 0.32.0 pypi_0 pypi
yaspin 2.1.0 pypi_0 pypi
zipp 3.6.0 pypi_0 pypi
zlib 1.2.11 h7f8727e_4 CUDA version and GPU Information
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Sun_Mar_21_19:15:46_PDT_2021
Cuda compilation tools, release 11.3, V11.3.58
Build cuda_11.3.r11.3/compiler.29745058_0
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.86 Driver Version: 470.86 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... Off | 00000000:01:00.0 Off | N/A |
| N/A 38C P0 9W / N/A | 9MiB / 3910MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1146 G /usr/lib/xorg/Xorg 4MiB |
| 0 N/A N/A 1939 G /usr/lib/xorg/Xorg 4MiB |
+-----------------------------------------------------------------------------+
OK. Could you add --device cpu to your command to make sure whether it is a device issue?
MendelXu
commented
2 years ago Wait a minute...If you are using the command python demo/image_demo.py planes.jpg configs/soft_teacher/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k.py work_dirs/iter_720000.pth --output work_dirs/, why it reported
tools/train.py FAILED Could you run it again?
vefak
commented
2 years ago Now I got following erros both with --device cpu and without --device cpu
After posting this issue, I only reinstalled mmcv.
(torch) vefak@vefak:~/SoftTeacher$ python demo/image_demo.py planes.jpg configs/soft_teacher/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k.py work_dirs/iter_720000.pth --output work_dirs/ --device cpu
/home/vefak/SoftTeacher/thirdparty/mmdetection/mmdet/core/anchor/builder.py:17: UserWarning: ``build_anchor_generator`` would be deprecated soon, please use ``build_prior_generator``
'``build_anchor_generator`` would be deprecated soon, please use '
Use load_from_local loader
It is not an error and just a warning which shouldn't affect the inference.
vefak
commented
2 years ago sorry i made a mistake. I actually made a mess. I was getting this error while trying to train. Here my actual and right command for training. I have only one GPU.
bash tools/dist_train.sh configs/soft_teacher/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k.py 1Again I am terribly sorry for mistake. As you said inference is okay
MendelXu
commented
2 years ago OK. I have no idea what happened. Could you try to run the baseline with
python -m torch.distributed.launch --nproc_per_node=1 --master_port=29995 tools/train.py configs/baseline/faster_rcnn_r50_caffe_fpn_coco_full_720k.py --launcher pytorchand
python tools/train.py configs/baseline/faster_rcnn_r50_caffe_fpn_coco_full_720k.py --gpus 1? If the first one doesn't work and the second one works, I can try to implement a non-distributed version.
vefak
commented
2 years ago In first one, I got same error which it is first post in this issue.
The second one I got different error again. Here:
Traceback (most recent call last):
File "tools/train.py", line 15, in <module>
from mmdet.models import build_detector
File "/home/vefak/SoftTeacher/thirdparty/mmdetection/mmdet/models/__init__.py", line 2, in <module>
from .backbones import * # noqa: F401,F403
File "/home/vefak/SoftTeacher/thirdparty/mmdetection/mmdet/models/backbones/__init__.py", line 2, in <module>
from .csp_darknet import CSPDarknet
File "/home/vefak/SoftTeacher/thirdparty/mmdetection/mmdet/models/backbones/csp_darknet.py", line 11, in <module>
from ..utils import CSPLayer
File "/home/vefak/SoftTeacher/thirdparty/mmdetection/mmdet/models/utils/__init__.py", line 12, in <module>
from .positional_encoding import (LearnedPositionalEncoding,
File "/home/vefak/SoftTeacher/thirdparty/mmdetection/mmdet/models/utils/positional_encoding.py", line 6, in <module>
from mmcv.cnn.bricks.transformer import POSITIONAL_ENCODING
File "/home/vefak/Documents/anaconda3/envs/torch/lib/python3.6/site-packages/mmcv/cnn/bricks/transformer.py", line 18, in <module>
from mmcv.ops.multi_scale_deform_attn import MultiScaleDeformableAttention # noqa F401
File "/home/vefak/Documents/anaconda3/envs/torch/lib/python3.6/site-packages/mmcv/ops/__init__.py", line 2, in <module>
from .assign_score_withk import assign_score_withk
File "/home/vefak/Documents/anaconda3/envs/torch/lib/python3.6/site-packages/mmcv/ops/assign_score_withk.py", line 6, in <module>
'_ext', ['assign_score_withk_forward', 'assign_score_withk_backward'])
File "/home/vefak/Documents/anaconda3/envs/torch/lib/python3.6/site-packages/mmcv/utils/ext_loader.py", line 15, in load_ext
assert hasattr(ext, fun), f'{fun} miss in module {name}'
AssertionError: assign_score_withk_forward miss in module _ext
It seems you haven't installed mmcv-full correctly. Maybe you can reinstall it.
vefak
commented
2 years ago Hi again, After installed everything again and installed mmcv from scratch. I tried both command you asked.
The below one I got similar error, ı guess.
python -m torch.distributed.launch --nproc_per_node=1 --master_port=29995 tools/train.py configs/baseline/faster_rcnn_r50_caffe_fpn_coco_full_720k.py --launcher pytorchThe module torch.distributed.launch is deprecated and going to be removed in future.Migrate to torch.distributed.run
WARNING:torch.distributed.run:--use_env is deprecated and will be removed in future releases.
Please read local_rank from `os.environ('LOCAL_RANK')` instead.
INFO:torch.distributed.launcher.api:Starting elastic_operator with launch configs:
entrypoint : tools/train.py
min_nodes : 1
max_nodes : 1
nproc_per_node : 1
run_id : none
rdzv_backend : static
rdzv_endpoint : 127.0.0.1:29995
rdzv_configs : {'rank': 0, 'timeout': 900}
max_restarts : 3
monitor_interval : 5
log_dir : None
metrics_cfg : {}
INFO:torch.distributed.elastic.agent.server.local_elastic_agent:log directory set to: /tmp/torchelastic_5474cwxi/none_43hbxbvu
INFO:torch.distributed.elastic.agent.server.api:[default] starting workers for entrypoint: python
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous'ing worker group
/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/torch/distributed/elastic/utils/store.py:53: FutureWarning: This is an experimental API and will be changed in future.
"This is an experimental API and will be changed in future.", FutureWarning
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result:
restart_count=0
master_addr=127.0.0.1
master_port=29995
group_rank=0
group_world_size=1
local_ranks=[0]
role_ranks=[0]
global_ranks=[0]
role_world_sizes=[1]
global_world_sizes=[1]
INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group
INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_5474cwxi/none_43hbxbvu/attempt_0/0/error.json
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda-11.3'
Traceback (most recent call last):
File "tools/train.py", line 198, in <module>
main()
File "tools/train.py", line 130, in main
init_dist(args.launcher, **cfg.dist_params)
File "/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/mmcv/runner/dist_utils.py", line 18, in init_dist
_init_dist_pytorch(backend, **kwargs)
File "/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/mmcv/runner/dist_utils.py", line 31, in _init_dist_pytorch
torch.cuda.set_device(rank % num_gpus)
ZeroDivisionError: integer division or modulo by zero
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 5867) of binary: /home/vefak/Documents/anaconda3/envs/teacher/bin/python
ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed
INFO:torch.distributed.elastic.agent.server.api:[default] Worker group FAILED. 3/3 attempts left; will restart worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Stopping worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous'ing worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result:
restart_count=1
master_addr=127.0.0.1
master_port=29995
group_rank=0
group_world_size=1
local_ranks=[0]
role_ranks=[0]
global_ranks=[0]
role_world_sizes=[1]
global_world_sizes=[1]
INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group
INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_5474cwxi/none_43hbxbvu/attempt_1/0/error.json
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda-11.3'
Traceback (most recent call last):
File "tools/train.py", line 198, in <module>
main()
File "tools/train.py", line 130, in main
init_dist(args.launcher, **cfg.dist_params)
File "/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/mmcv/runner/dist_utils.py", line 18, in init_dist
_init_dist_pytorch(backend, **kwargs)
File "/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/mmcv/runner/dist_utils.py", line 31, in _init_dist_pytorch
torch.cuda.set_device(rank % num_gpus)
ZeroDivisionError: integer division or modulo by zero
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 5879) of binary: /home/vefak/Documents/anaconda3/envs/teacher/bin/python
ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed
INFO:torch.distributed.elastic.agent.server.api:[default] Worker group FAILED. 2/3 attempts left; will restart worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Stopping worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous'ing worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result:
restart_count=2
master_addr=127.0.0.1
master_port=29995
group_rank=0
group_world_size=1
local_ranks=[0]
role_ranks=[0]
global_ranks=[0]
role_world_sizes=[1]
global_world_sizes=[1]
INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group
INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_5474cwxi/none_43hbxbvu/attempt_2/0/error.json
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda-11.3'
Traceback (most recent call last):
File "tools/train.py", line 198, in <module>
main()
File "tools/train.py", line 130, in main
init_dist(args.launcher, **cfg.dist_params)
File "/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/mmcv/runner/dist_utils.py", line 18, in init_dist
_init_dist_pytorch(backend, **kwargs)
File "/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/mmcv/runner/dist_utils.py", line 31, in _init_dist_pytorch
torch.cuda.set_device(rank % num_gpus)
ZeroDivisionError: integer division or modulo by zero
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 5891) of binary: /home/vefak/Documents/anaconda3/envs/teacher/bin/python
ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed
INFO:torch.distributed.elastic.agent.server.api:[default] Worker group FAILED. 1/3 attempts left; will restart worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Stopping worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous'ing worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result:
restart_count=3
master_addr=127.0.0.1
master_port=29995
group_rank=0
group_world_size=1
local_ranks=[0]
role_ranks=[0]
global_ranks=[0]
role_world_sizes=[1]
global_world_sizes=[1]
INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group
INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_5474cwxi/none_43hbxbvu/attempt_3/0/error.json
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda-11.3'
Traceback (most recent call last):
File "tools/train.py", line 198, in <module>
main()
File "tools/train.py", line 130, in main
init_dist(args.launcher, **cfg.dist_params)
File "/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/mmcv/runner/dist_utils.py", line 18, in init_dist
_init_dist_pytorch(backend, **kwargs)
File "/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/mmcv/runner/dist_utils.py", line 31, in _init_dist_pytorch
torch.cuda.set_device(rank % num_gpus)
ZeroDivisionError: integer division or modulo by zero
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 5904) of binary: /home/vefak/Documents/anaconda3/envs/teacher/bin/python
ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed
INFO:torch.distributed.elastic.agent.server.api:Local worker group finished (FAILED). Waiting 300 seconds for other agents to finish
/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/torch/distributed/elastic/utils/store.py:71: FutureWarning: This is an experimental API and will be changed in future.
"This is an experimental API and will be changed in future.", FutureWarning
INFO:torch.distributed.elastic.agent.server.api:Done waiting for other agents. Elapsed: 0.00039005279541015625 seconds
{"name": "torchelastic.worker.status.FAILED", "source": "WORKER", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": 0, "group_rank": 0, "worker_id": "5904", "role": "default", "hostname": "vefak", "state": "FAILED", "total_run_time": 20, "rdzv_backend": "static", "raw_error": "{\"message\": \"<NONE>\"}", "metadata": "{\"group_world_size\": 1, \"entry_point\": \"python\", \"local_rank\": [0], \"role_rank\": [0], \"role_world_size\": [1]}", "agent_restarts": 3}}
{"name": "torchelastic.worker.status.SUCCEEDED", "source": "AGENT", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": null, "group_rank": 0, "worker_id": null, "role": "default", "hostname": "vefak", "state": "SUCCEEDED", "total_run_time": 20, "rdzv_backend": "static", "raw_error": null, "metadata": "{\"group_world_size\": 1, \"entry_point\": \"python\"}", "agent_restarts": 3}}
/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py:354: UserWarning:
**********************************************************************
CHILD PROCESS FAILED WITH NO ERROR_FILE
**********************************************************************
CHILD PROCESS FAILED WITH NO ERROR_FILE
Child process 5904 (local_rank 0) FAILED (exitcode 1)
Error msg: Process failed with exitcode 1
Without writing an error file to <N/A>.
While this DOES NOT affect the correctness of your application,
no trace information about the error will be available for inspection.
Consider decorating your top level entrypoint function with
torch.distributed.elastic.multiprocessing.errors.record. Example:
from torch.distributed.elastic.multiprocessing.errors import record
@record
def trainer_main(args):
# do train
**********************************************************************
warnings.warn(_no_error_file_warning_msg(rank, failure))
Traceback (most recent call last):
File "/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/torch/distributed/launch.py", line 173, in <module>
main()
File "/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/torch/distributed/launch.py", line 169, in main
run(args)
File "/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/torch/distributed/run.py", line 624, in run
)(*cmd_args)
File "/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/torch/distributed/launcher/api.py", line 116, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 348, in wrapper
return f(*args, **kwargs)
File "/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/torch/distributed/launcher/api.py", line 247, in launch_agent
failures=result.failures,
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
***************************************
tools/train.py FAILED
=======================================
Root Cause:
[0]:
time: 2022-02-14_21:24:59
rank: 0 (local_rank: 0)
exitcode: 1 (pid: 5904)
error_file: <N/A>
msg: "Process failed with exitcode 1"
=======================================
Other Failures:
<NO_OTHER_FAILURES>
***************************************
The second one, I got different error but I can count it as progress. This time models and annotations are loaded. I saw that it is about using non-distributed mode. Mentioned in #37. What should I do? I cannot use normal mode and non-distributed mode raised different error. The second command and its output
python tools/train.py configs/baseline/faster_rcnn_r50_caffe_fpn_coco_full_720k.py --gpus 1
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda-11.3'
2022-02-14 00:07:51,433 - mmdet.ssod - INFO - [<StreamHandler <stderr> (INFO)>, <FileHandler /home/vefak/Documents/SoftTeacher/work_dirs/faster_rcnn_r50_caffe_fpn_coco_full_720k/20220214_000751.log (INFO)>]
2022-02-14 00:07:51,434 - mmdet.ssod - INFO - Environment info:
------------------------------------------------------------
sys.platform: linux
Python: 3.6.13 |Anaconda, Inc.| (default, Jun 4 2021, 14:25:59) [GCC 7.5.0]
CUDA available: False
GCC: gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
PyTorch: 1.9.0
PyTorch compiling details: PyTorch built with:
- GCC 7.3
- C++ Version: 201402
- Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v2.1.2 (Git Hash 98be7e8afa711dc9b66c8ff3504129cb82013cdb)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- NNPACK is enabled
- CPU capability usage: AVX2
- Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.9.0, USE_CUDA=0, USE_CUDNN=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON,
TorchVision: 0.10.0
OpenCV: 4.5.5
MMCV: 1.3.16
MMCV Compiler: GCC 9.3
MMCV CUDA Compiler: not available
MMDetection: 2.16.0+bef9a25
------------------------------------------------------------
2022-02-14 00:07:51,882 - mmdet.ssod - INFO - Distributed training: False
2022-02-14 00:07:52,332 - mmdet.ssod - INFO - Config:
model = dict(
type='FasterRCNN',
backbone=dict(
type='ResNet',
depth=50,
num_stages=4,
out_indices=(0, 1, 2, 3),
frozen_stages=1,
norm_cfg=dict(type='BN', requires_grad=False),
norm_eval=True,
style='caffe',
init_cfg=dict(
type='Pretrained',
checkpoint='open-mmlab://detectron2/resnet50_caffe')),
neck=dict(
type='FPN',
in_channels=[256, 512, 1024, 2048],
out_channels=256,
num_outs=5),
rpn_head=dict(
type='RPNHead',
in_channels=256,
feat_channels=256,
anchor_generator=dict(
type='AnchorGenerator',
scales=[8],
ratios=[0.5, 1.0, 2.0],
strides=[4, 8, 16, 32, 64]),
bbox_coder=dict(
type='DeltaXYWHBBoxCoder',
target_means=[0.0, 0.0, 0.0, 0.0],
target_stds=[1.0, 1.0, 1.0, 1.0]),
loss_cls=dict(
type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0),
loss_bbox=dict(type='L1Loss', loss_weight=1.0)),
roi_head=dict(
type='StandardRoIHead',
bbox_roi_extractor=dict(
type='SingleRoIExtractor',
roi_layer=dict(type='RoIAlign', output_size=7, sampling_ratio=0),
out_channels=256,
featmap_strides=[4, 8, 16, 32]),
bbox_head=dict(
type='Shared2FCBBoxHead',
in_channels=256,
fc_out_channels=1024,
roi_feat_size=7,
num_classes=80,
bbox_coder=dict(
type='DeltaXYWHBBoxCoder',
target_means=[0.0, 0.0, 0.0, 0.0],
target_stds=[0.1, 0.1, 0.2, 0.2]),
reg_class_agnostic=False,
loss_cls=dict(
type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0),
loss_bbox=dict(type='L1Loss', loss_weight=1.0))),
train_cfg=dict(
rpn=dict(
assigner=dict(
type='MaxIoUAssigner',
pos_iou_thr=0.7,
neg_iou_thr=0.3,
min_pos_iou=0.3,
match_low_quality=True,
ignore_iof_thr=-1),
sampler=dict(
type='RandomSampler',
num=256,
pos_fraction=0.5,
neg_pos_ub=-1,
add_gt_as_proposals=False),
allowed_border=-1,
pos_weight=-1,
debug=False),
rpn_proposal=dict(
nms_pre=2000,
max_per_img=1000,
nms=dict(type='nms', iou_threshold=0.7),
min_bbox_size=0),
rcnn=dict(
assigner=dict(
type='MaxIoUAssigner',
pos_iou_thr=0.5,
neg_iou_thr=0.5,
min_pos_iou=0.5,
match_low_quality=False,
ignore_iof_thr=-1),
sampler=dict(
type='RandomSampler',
num=512,
pos_fraction=0.25,
neg_pos_ub=-1,
add_gt_as_proposals=True),
pos_weight=-1,
debug=False)),
test_cfg=dict(
rpn=dict(
nms_pre=1000,
max_per_img=1000,
nms=dict(type='nms', iou_threshold=0.7),
min_bbox_size=0),
rcnn=dict(
score_thr=0.05,
nms=dict(type='nms', iou_threshold=0.5),
max_per_img=100)))
dataset_type = 'CocoDataset'
data_root = 'data/coco/'
img_norm_cfg = dict(
mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False)
train_pipeline = [
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations', with_bbox=True),
dict(
type='Sequential',
transforms=[
dict(
type='RandResize',
img_scale=[(1333, 400), (1333, 1200)],
multiscale_mode='range',
keep_ratio=True),
dict(type='RandFlip', flip_ratio=0.5),
dict(
type='OneOf',
transforms=[
dict(type='Identity'),
dict(type='AutoContrast'),
dict(type='RandEqualize'),
dict(type='RandSolarize'),
dict(type='RandColor'),
dict(type='RandContrast'),
dict(type='RandBrightness'),
dict(type='RandSharpness'),
dict(type='RandPosterize')
])
]),
dict(type='Pad', size_divisor=32),
dict(
type='Normalize',
mean=[103.53, 116.28, 123.675],
std=[1.0, 1.0, 1.0],
to_rgb=False),
dict(type='ExtraAttrs', tag='sup'),
dict(type='DefaultFormatBundle'),
dict(
type='Collect',
keys=['img', 'gt_bboxes', 'gt_labels'],
meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg',
'pad_shape', 'scale_factor', 'tag'))
]
test_pipeline = [
dict(type='LoadImageFromFile'),
dict(
type='MultiScaleFlipAug',
img_scale=(1333, 800),
flip=False,
transforms=[
dict(type='Resize', keep_ratio=True),
dict(type='RandomFlip'),
dict(
type='Normalize',
mean=[103.53, 116.28, 123.675],
std=[1.0, 1.0, 1.0],
to_rgb=False),
dict(type='Pad', size_divisor=32),
dict(type='ImageToTensor', keys=['img']),
dict(type='Collect', keys=['img'])
])
]
data = dict(
samples_per_gpu=2,
workers_per_gpu=2,
train=dict(
type='CocoDataset',
ann_file='data/coco/annotations/instances_train2017.json',
img_prefix='data/coco/train2017/',
pipeline=[
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations', with_bbox=True),
dict(
type='Sequential',
transforms=[
dict(
type='RandResize',
img_scale=[(1333, 400), (1333, 1200)],
multiscale_mode='range',
keep_ratio=True),
dict(type='RandFlip', flip_ratio=0.5),
dict(
type='OneOf',
transforms=[
dict(type='Identity'),
dict(type='AutoContrast'),
dict(type='RandEqualize'),
dict(type='RandSolarize'),
dict(type='RandColor'),
dict(type='RandContrast'),
dict(type='RandBrightness'),
dict(type='RandSharpness'),
dict(type='RandPosterize')
])
]),
dict(type='Pad', size_divisor=32),
dict(
type='Normalize',
mean=[103.53, 116.28, 123.675],
std=[1.0, 1.0, 1.0],
to_rgb=False),
dict(type='ExtraAttrs', tag='sup'),
dict(type='DefaultFormatBundle'),
dict(
type='Collect',
keys=['img', 'gt_bboxes', 'gt_labels'],
meta_keys=('filename', 'ori_shape', 'img_shape',
'img_norm_cfg', 'pad_shape', 'scale_factor', 'tag'))
]),
val=dict(
type='CocoDataset',
ann_file='data/coco/annotations/instances_val2017.json',
img_prefix='data/coco/val2017/',
pipeline=[
dict(type='LoadImageFromFile'),
dict(
type='MultiScaleFlipAug',
img_scale=(1333, 800),
flip=False,
transforms=[
dict(type='Resize', keep_ratio=True),
dict(type='RandomFlip'),
dict(
type='Normalize',
mean=[103.53, 116.28, 123.675],
std=[1.0, 1.0, 1.0],
to_rgb=False),
dict(type='Pad', size_divisor=32),
dict(type='ImageToTensor', keys=['img']),
dict(type='Collect', keys=['img'])
])
]),
test=dict(
type='CocoDataset',
ann_file='data/coco/annotations/instances_val2017.json',
img_prefix='data/coco/val2017/',
pipeline=[
dict(type='LoadImageFromFile'),
dict(
type='MultiScaleFlipAug',
img_scale=(1333, 800),
flip=False,
transforms=[
dict(type='Resize', keep_ratio=True),
dict(type='RandomFlip'),
dict(
type='Normalize',
mean=[103.53, 116.28, 123.675],
std=[1.0, 1.0, 1.0],
to_rgb=False),
dict(type='Pad', size_divisor=32),
dict(type='ImageToTensor', keys=['img']),
dict(type='Collect', keys=['img'])
])
]))
evaluation = dict(interval=4000, metric='bbox')
optimizer = dict(type='SGD', lr=0.02, momentum=0.9, weight_decay=0.0001)
optimizer_config = dict(grad_clip=None)
lr_config = dict(
policy='step',
warmup='linear',
warmup_iters=500,
warmup_ratio=0.001,
step=[480000, 640000])
runner = dict(type='IterBasedRunner', max_iters=720000)
checkpoint_config = dict(interval=4000, by_epoch=False, max_keep_ckpts=10)
log_config = dict(
interval=50,
hooks=[
dict(type='TextLoggerHook', by_epoch=False),
dict(
type='WandbLoggerHook',
init_kwargs=dict(
project='pre_release',
name='faster_rcnn_r50_caffe_fpn_coco_full_720k',
config=dict(
work_dirs=
'./work_dirs/faster_rcnn_r50_caffe_fpn_coco_full_720k',
total_step=720000)),
by_epoch=False)
])
custom_hooks = [dict(type='NumClassCheckHook')]
dist_params = dict(backend='nccl')
log_level = 'INFO'
load_from = None
resume_from = None
workflow = [('train', 1)]
mmdet_base = '../../thirdparty/mmdetection/configs/_base_'
fp16 = dict(loss_scale='dynamic')
work_dir = './work_dirs/faster_rcnn_r50_caffe_fpn_coco_full_720k'
cfg_name = 'faster_rcnn_r50_caffe_fpn_coco_full_720k'
gpu_ids = range(0, 1)
/home/vefak/Documents/SoftTeacher/thirdparty/mmdetection/mmdet/core/anchor/builder.py:17: UserWarning: ``build_anchor_generator`` would be deprecated soon, please use ``build_prior_generator``
'``build_anchor_generator`` would be deprecated soon, please use '
2022-02-14 00:07:52,754 - mmdet.ssod - INFO - initialize ResNet with init_cfg {'type': 'Pretrained', 'checkpoint': 'open-mmlab://detectron2/resnet50_caffe'}
2022-02-14 00:07:52,755 - mmcv - INFO - load model from: open-mmlab://detectron2/resnet50_caffe
2022-02-14 00:07:52,755 - mmcv - INFO - Use load_from_openmmlab loader
2022-02-14 00:07:52,840 - mmcv - WARNING - The model and loaded state dict do not match exactly
unexpected key in source state_dict: conv1.bias
2022-02-14 00:07:52,854 - mmdet.ssod - INFO - initialize FPN with init_cfg {'type': 'Xavier', 'layer': 'Conv2d', 'distribution': 'uniform'}
2022-02-14 00:07:52,880 - mmdet.ssod - INFO - initialize RPNHead with init_cfg {'type': 'Normal', 'layer': 'Conv2d', 'std': 0.01}
2022-02-14 00:07:52,884 - mmdet.ssod - INFO - initialize Shared2FCBBoxHead with init_cfg [{'type': 'Normal', 'std': 0.01, 'override': {'name': 'fc_cls'}}, {'type': 'Normal', 'std': 0.001, 'override': {'name': 'fc_reg'}}, {'type': 'Xavier', 'layer': 'Linear', 'override': [{'name': 'shared_fcs'}, {'name': 'cls_fcs'}, {'name': 'reg_fcs'}]}]
loading annotations into memory...
Done (t=14.22s)
creating index...
index created!
Traceback (most recent call last):
File "tools/train.py", line 198, in <module>
main()
File "tools/train.py", line 193, in main
meta=meta,
File "/home/vefak/Documents/SoftTeacher/ssod/apis/train.py", line 81, in train_detector
for ds in dataset
File "/home/vefak/Documents/SoftTeacher/ssod/apis/train.py", line 81, in <listcomp>
for ds in dataset
File "/home/vefak/Documents/SoftTeacher/ssod/datasets/builder.py", line 69, in build_dataloader
if shuffle
File "/home/vefak/Documents/SoftTeacher/ssod/datasets/builder.py", line 40, in build_sampler
return build_from_cfg(cfg, SAMPLERS, default_args)
File "/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/mmcv/utils/registry.py", line 45, in build_from_cfg
f'{obj_type} is not in the {registry.name} registry')
KeyError: 'Sampler is not in the sampler registry'Hi again, Is there any updates for non distrubed version?
vefak
commented
2 years ago The problem is about incompability between RTX3050, pytorch and mmcv-full RTX3050 doesnot support CUDA 10.x version. It should be specific configuration between them.
Currently I installed CUDA 11.1 Pytorch 1.9.0 mmcv 1.3.9
tanjary21
commented
2 years ago Hi there, I'm running into this issue if my annotation.json file is huge(7Gb).
I am able to train my model on multiple GPUs if I use a smaller slice of the full dataset using the command:
!CUDA_VISIBLE_DEVICES=0,1,2,3 PORT=29501 ./tools/dist_train.sh ./configs/motsynth/qdtrack_frcnn_r50_fpn_4e_motsynth.py 4 --work-dir work_dirs/MOTSynth/virgin --cfg-options 'optimizer.lr=0.01' 'data.train.ann_file=data/MOTSynth/annotations/test_cocoformat.json',
and I can train it on the full dataset when I use a single GPU using the following command:
!python ./tools/train.py ./configs/motsynth/qdtrack_frcnn_r50_fpn_4e_motsynth.py --cfg-options work_dir="work_dirs/MOTSynth/virgin" optimizer.lr=0.0025.
My environment has the following packages:
hitbuyi
commented
5 months ago I have the same issue, what's going on this problem?
The module torch.distributed.launch is deprecated and going to be removed in future.Migrate to torch.distributed.run WARNING:torch.distributed.run:--use_env is deprecated and will be removed in future releases. Please read local_rank fromos.environ('LOCAL_RANK')` instead. INFO:torch.distributed.launcher.api:Starting elastic_operator with launch configs: entrypoint : tools/train.py min_nodes : 1 max_nodes : 1 nproc_per_node : 1 run_id : none rdzv_backend : static rdzv_endpoint : 127.0.0.1:29500 rdzv_configs : {'rank': 0, 'timeout': 900} max_restarts : 3 monitor_interval : 5 log_dir : None metrics_cfg : {}INFO:torch.distributed.elastic.agent.server.local_elastic_agent:log directory set to: /tmp/torchelastic_mnn5x9jo/none_j57hpvun INFO:torch.distributed.elastic.agent.server.api:[default] starting workers for entrypoint: python3 INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous'ing worker group /home/vefak/Documents/anaconda3/envs/torch/lib/python3.6/site-packages/torch/distributed/elastic/utils/store.py:53: FutureWarning: This is an experimental API and will be changed in future. "This is an experimental API and will be changed in future.", FutureWarning INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result: restart_count=0 master_addr=127.0.0.1 master_port=29500 group_rank=0 group_world_size=1 local_ranks=[0] role_ranks=[0] global_ranks=[0] role_world_sizes=[1] global_world_sizes=[1]
INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_mnn5x9jo/none_j57hpvun/attempt_0/0/error.json ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -11) local_rank: 0 (pid: 14927) of binary: /home/vefak/Documents/anaconda3/envs/torch/bin/python3 ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed INFO:torch.distributed.elastic.agent.server.api:[default] Worker group FAILED. 3/3 attempts left; will restart worker group INFO:torch.distributed.elastic.agent.server.api:[default] Stopping worker group INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous'ing worker group INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result: restart_count=1 master_addr=127.0.0.1 master_port=29500 group_rank=0 group_world_size=1 local_ranks=[0] role_ranks=[0] global_ranks=[0] role_world_sizes=[1] global_world_sizes=[1]
INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_mnn5x9jo/none_j57hpvun/attempt_1/0/error.json ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -11) local_rank: 0 (pid: 14955) of binary: /home/vefak/Documents/anaconda3/envs/torch/bin/python3 ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed INFO:torch.distributed.elastic.agent.server.api:[default] Worker group FAILED. 2/3 attempts left; will restart worker group INFO:torch.distributed.elastic.agent.server.api:[default] Stopping worker group INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous'ing worker group INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result: restart_count=2 master_addr=127.0.0.1 master_port=29500 group_rank=0 group_world_size=1 local_ranks=[0] role_ranks=[0] global_ranks=[0] role_world_sizes=[1] global_world_sizes=[1]
INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_mnn5x9jo/none_j57hpvun/attempt_2/0/error.json ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -11) local_rank: 0 (pid: 15007) of binary: /home/vefak/Documents/anaconda3/envs/torch/bin/python3 ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed INFO:torch.distributed.elastic.agent.server.api:[default] Worker group FAILED. 1/3 attempts left; will restart worker group INFO:torch.distributed.elastic.agent.server.api:[default] Stopping worker group INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous'ing worker group INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result: restart_count=3 master_addr=127.0.0.1 master_port=29500 group_rank=0 group_world_size=1 local_ranks=[0] role_ranks=[0] global_ranks=[0] role_world_sizes=[1] global_world_sizes=[1]
INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_mnn5x9jo/none_j57hpvun/attempt_3/0/error.json ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -11) local_rank: 0 (pid: 15048) of binary: /home/vefak/Documents/anaconda3/envs/torch/bin/python3 ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed INFO:torch.distributed.elastic.agent.server.api:Local worker group finished (FAILED). Waiting 300 seconds for other agents to finish /home/vefak/Documents/anaconda3/envs/torch/lib/python3.6/site-packages/torch/distributed/elastic/utils/store.py:71: FutureWarning: This is an experimental API and will be changed in future. "This is an experimental API and will be changed in future.", FutureWarning INFO:torch.distributed.elastic.agent.server.api:Done waiting for other agents. Elapsed: 0.0004889965057373047 seconds {"name": "torchelastic.worker.status.FAILED", "source": "WORKER", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": 0, "group_rank": 0, "worker_id": "15048", "role": "default", "hostname": "vefak", "state": "FAILED", "total_run_time": 25, "rdzv_backend": "static", "raw_error": "{\"message\": \"\"}", "metadata": "{\"group_world_size\": 1, \"entry_point\": \"python3\", \"local_rank\": [0], \"role_rank\": [0], \"role_world_size\": [1]}", "agent_restarts": 3}}
{"name": "torchelastic.worker.status.SUCCEEDED", "source": "AGENT", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": null, "group_rank": 0, "worker_id": null, "role": "default", "hostname": "vefak", "state": "SUCCEEDED", "total_run_time": 25, "rdzv_backend": "static", "raw_error": null, "metadata": "{\"group_world_size\": 1, \"entry_point\": \"python3\"}", "agent_restarts": 3}}
/home/vefak/Documents/anaconda3/envs/torch/lib/python3.6/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py:354: UserWarning:
CHILD PROCESS FAILED WITH NO ERROR_FILE Child process 15048 (local_rank 0) FAILED (exitcode -11) Error msg: Signal 11 (SIGSEGV) received by PID 15048 Without writing an error file to <N/A>. While this DOES NOT affect the correctness of your application, no trace information about the error will be available for inspection. Consider decorating your top level entrypoint function with torch.distributed.elastic.multiprocessing.errors.record. Example:
from torch.distributed.elastic.multiprocessing.errors import record
@record def trainer_main(args):
do train
warnings.warn(_no_error_file_warning_msg(rank, failure)) Traceback (most recent call last): File "/home/vefak/Documents/anaconda3/envs/torch/lib/python3.6/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/home/vefak/Documents/anaconda3/envs/torch/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/vefak/Documents/anaconda3/envs/torch/lib/python3.6/site-packages/torch/distributed/launch.py", line 173, in
main()
File "/home/vefak/Documents/anaconda3/envs/torch/lib/python3.6/site-packages/torch/distributed/launch.py", line 169, in main
run(args)
File "/home/vefak/Documents/anaconda3/envs/torch/lib/python3.6/site-packages/torch/distributed/run.py", line 624, in run
)(cmd_args)
File "/home/vefak/Documents/anaconda3/envs/torch/lib/python3.6/site-packages/torch/distributed/launcher/api.py", line 116, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/vefak/Documents/anaconda3/envs/torch/lib/python3.6/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 348, in wrapper
return f(args, **kwargs)
File "/home/vefak/Documents/anaconda3/envs/torch/lib/python3.6/site-packages/torch/distributed/launcher/api.py", line 247, in launch_agent
failures=result.failures,
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
================================================== Root Cause: [0]: time: 2022-02-04_00:19:07 rank: 0 (local_rank: 0) exitcode: -11 (pid: 15048) error_file: <N/A> msg: "Signal 11 (SIGSEGV) received by PID 15048"
Other Failures: