Closed vefak closed 2 years ago
Sorry for the late reply. I just spent a long vacation. What is your command to run the job?
Sorry for the late reply. I just spent a long vacation. What is your command to run the job?
Hi thanks for replying. Here my command
python demo/image_demo.py planes.jpg configs/soft_teacher/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k.py work_dirs/iter_720000.pth --output work_dirs/
Are you executing it on WSL? Could you add some details like operating system, cuda version.
Are you executing it on WSL? Could you add some details like operating system, cuda version.
No I dont. Here my system information. And I am running on conda env
Distributor ID: Ubuntu
Description: Ubuntu 20.04.3 LTS
Release: 20.04
Codename: focal
Here my conda env packages (Conda 4.10.3)
packages in environment at /home/vefak/Documents/anaconda3/envs/torch:
Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 4.5 1_gnu
addict 2.4.0 pypi_0 pypi
bzip2 1.0.8 h7f98852_4 conda-forge
ca-certificates 2021.10.26 h06a4308_2
certifi 2021.5.30 py36h5fab9bb_0 conda-forge
cffi 1.14.6 py36hc120d54_0 conda-forge
charset-normalizer 2.0.11 pypi_0 pypi
click 8.0.3 pypi_0 pypi
configparser 5.2.0 pypi_0 pypi
cpuonly 1.0 0 pytorch
cudatoolkit 11.3.1 h2bc3f7f_2
cudnn 8.2.1.32 h86fa8c9_0 conda-forge
cycler 0.11.0 pypi_0 pypi
dataclasses 0.8 pyh787bdff_2 conda-forge
docker-pycreds 0.4.0 pypi_0 pypi
ffmpeg 4.3 hf484d3e_0 pytorch
freetype 2.10.4 h0708190_1 conda-forge
future 0.18.2 py36h5fab9bb_3 conda-forge
gitdb 4.0.9 pypi_0 pypi
gitpython 3.1.18 pypi_0 pypi
gmp 6.2.1 h58526e2_0 conda-forge
gnutls 3.6.13 h85f3911_1 conda-forge
idna 3.3 pypi_0 pypi
importlib-metadata 4.8.3 pypi_0 pypi
install 1.3.5 pypi_0 pypi
intel-openmp 2021.4.0 h06a4308_3561
jpeg 9b h024ee3a_2
kiwisolver 1.3.1 pypi_0 pypi
lame 3.100 h7f98852_1001 conda-forge
ld_impl_linux-64 2.35.1 h7274673_9
libblas 3.9.0 12_linux64_mkl conda-forge
libcblas 3.9.0 12_linux64_mkl conda-forge
libffi 3.3 he6710b0_2
libgcc-ng 9.3.0 h5101ec6_17
libgomp 9.3.0 h5101ec6_17
libiconv 1.16 h516909a_0 conda-forge
liblapack 3.9.0 12_linux64_mkl conda-forge
libpng 1.6.37 h21135ba_2 conda-forge
libprotobuf 3.16.0 h780b84a_0 conda-forge
libstdcxx-ng 9.3.0 hd4cf53a_17
libtiff 4.0.9 he6b73bb_1 conda-forge
magma 2.5.4 h6103c52_2 conda-forge
matplotlib 3.3.4 pypi_0 pypi
mkl 2021.4.0 h06a4308_640
mmcv-full 1.3.9 dev_0 <develop>
mmdet 2.16.0 dev_0 <develop>
nccl 2.11.4.1 hdc17891_0 conda-forge
ncurses 6.3 h7f8727e_2
nettle 3.6 he412f7d_0 conda-forge
ninja 1.10.2 h4bd325d_0 conda-forge
numpy 1.19.5 py36hfc0c790_2 conda-forge
olefile 0.46 pyh9f0ad1d_1 conda-forge
opencv-python 4.5.5.62 pypi_0 pypi
openh264 2.1.1 h780b84a_0 conda-forge
openssl 1.1.1m h7f8727e_0
packaging 21.3 pypi_0 pypi
pathtools 0.1.2 pypi_0 pypi
pillow 8.4.0 pypi_0 pypi
pip 21.2.2 py36h06a4308_0
prettytable 2.5.0 pypi_0 pypi
promise 2.3 pypi_0 pypi
protobuf 3.19.4 pypi_0 pypi
psutil 5.9.0 pypi_0 pypi
pycocotools 2.0.4 pypi_0 pypi
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pyparsing 3.0.7 pypi_0 pypi
python 3.6.13 h12debd9_1
python-dateutil 2.8.2 pypi_0 pypi
python_abi 3.6 2_cp36m conda-forge
pytorch 1.9.0 cuda112py36h755b813_1 conda-forge
pytorch-gpu 1.9.0 cuda112py36h0bbbad9_1 conda-forge
pyyaml 6.0 pypi_0 pypi
readline 8.1.2 h7f8727e_1
requests 2.27.1 pypi_0 pypi
sentry-sdk 1.5.4 pypi_0 pypi
setuptools 58.0.4 py36h06a4308_0
shortuuid 1.0.8 pypi_0 pypi
six 1.16.0 pypi_0 pypi
sleef 3.5.1 h7f98852_1 conda-forge
smmap 5.0.0 pypi_0 pypi
sqlite 3.37.0 hc218d9a_0
ssod 0.0.1 dev_0 <develop>
subprocess32 3.5.4 pypi_0 pypi
termcolor 1.1.0 pypi_0 pypi
terminaltables 3.1.10 pypi_0 pypi
tk 8.6.11 h1ccaba5_0
torchaudio 0.9.0 py36 pytorch
torchvision 0.10.0 py36_cpu [cpuonly] pytorch
typing_extensions 4.0.1 pyha770c72_0 conda-forge
urllib3 1.26.8 pypi_0 pypi
wandb 0.10.31 pypi_0 pypi
wcwidth 0.2.5 pypi_0 pypi
wheel 0.37.1 pyhd3eb1b0_0
xz 5.2.5 h7b6447c_0
yapf 0.32.0 pypi_0 pypi
yaspin 2.1.0 pypi_0 pypi
zipp 3.6.0 pypi_0 pypi
zlib 1.2.11 h7f8727e_4
CUDA version and GPU Information
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Sun_Mar_21_19:15:46_PDT_2021
Cuda compilation tools, release 11.3, V11.3.58
Build cuda_11.3.r11.3/compiler.29745058_0
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.86 Driver Version: 470.86 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... Off | 00000000:01:00.0 Off | N/A |
| N/A 38C P0 9W / N/A | 9MiB / 3910MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1146 G /usr/lib/xorg/Xorg 4MiB |
| 0 N/A N/A 1939 G /usr/lib/xorg/Xorg 4MiB |
+-----------------------------------------------------------------------------+
OK. Could you add --device cpu
to your command to make sure whether it is a device issue?
Wait a minute...If you are using the command python demo/image_demo.py planes.jpg configs/soft_teacher/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k.py work_dirs/iter_720000.pth --output work_dirs/
, why it reported
tools/train.py FAILED
Could you run it again?
Now I got following erros both with --device cpu
and without --device cpu
After posting this issue, I only reinstalled mmcv.
(torch) vefak@vefak:~/SoftTeacher$ python demo/image_demo.py planes.jpg configs/soft_teacher/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k.py work_dirs/iter_720000.pth --output work_dirs/ --device cpu
/home/vefak/SoftTeacher/thirdparty/mmdetection/mmdet/core/anchor/builder.py:17: UserWarning: ``build_anchor_generator`` would be deprecated soon, please use ``build_prior_generator``
'``build_anchor_generator`` would be deprecated soon, please use '
Use load_from_local loader
It is not an error and just a warning which shouldn't affect the inference.
sorry i made a mistake. I actually made a mess. I was getting this error while trying to train. Here my actual and right command for training. I have only one GPU.
bash tools/dist_train.sh configs/soft_teacher/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k.py 1
Again I am terribly sorry for mistake. As you said inference is okay
OK. I have no idea what happened. Could you try to run the baseline with
python -m torch.distributed.launch --nproc_per_node=1 --master_port=29995 tools/train.py configs/baseline/faster_rcnn_r50_caffe_fpn_coco_full_720k.py --launcher pytorch
and
python tools/train.py configs/baseline/faster_rcnn_r50_caffe_fpn_coco_full_720k.py --gpus 1
? If the first one doesn't work and the second one works, I can try to implement a non-distributed version.
In first one, I got same error which it is first post in this issue.
The second one I got different error again. Here:
Traceback (most recent call last):
File "tools/train.py", line 15, in <module>
from mmdet.models import build_detector
File "/home/vefak/SoftTeacher/thirdparty/mmdetection/mmdet/models/__init__.py", line 2, in <module>
from .backbones import * # noqa: F401,F403
File "/home/vefak/SoftTeacher/thirdparty/mmdetection/mmdet/models/backbones/__init__.py", line 2, in <module>
from .csp_darknet import CSPDarknet
File "/home/vefak/SoftTeacher/thirdparty/mmdetection/mmdet/models/backbones/csp_darknet.py", line 11, in <module>
from ..utils import CSPLayer
File "/home/vefak/SoftTeacher/thirdparty/mmdetection/mmdet/models/utils/__init__.py", line 12, in <module>
from .positional_encoding import (LearnedPositionalEncoding,
File "/home/vefak/SoftTeacher/thirdparty/mmdetection/mmdet/models/utils/positional_encoding.py", line 6, in <module>
from mmcv.cnn.bricks.transformer import POSITIONAL_ENCODING
File "/home/vefak/Documents/anaconda3/envs/torch/lib/python3.6/site-packages/mmcv/cnn/bricks/transformer.py", line 18, in <module>
from mmcv.ops.multi_scale_deform_attn import MultiScaleDeformableAttention # noqa F401
File "/home/vefak/Documents/anaconda3/envs/torch/lib/python3.6/site-packages/mmcv/ops/__init__.py", line 2, in <module>
from .assign_score_withk import assign_score_withk
File "/home/vefak/Documents/anaconda3/envs/torch/lib/python3.6/site-packages/mmcv/ops/assign_score_withk.py", line 6, in <module>
'_ext', ['assign_score_withk_forward', 'assign_score_withk_backward'])
File "/home/vefak/Documents/anaconda3/envs/torch/lib/python3.6/site-packages/mmcv/utils/ext_loader.py", line 15, in load_ext
assert hasattr(ext, fun), f'{fun} miss in module {name}'
AssertionError: assign_score_withk_forward miss in module _ext
It seems you haven't installed mmcv-full
correctly. Maybe you can reinstall it.
Hi again, After installed everything again and installed mmcv from scratch. I tried both command you asked.
The below one I got similar error, ı guess.
python -m torch.distributed.launch --nproc_per_node=1 --master_port=29995 tools/train.py configs/baseline/faster_rcnn_r50_caffe_fpn_coco_full_720k.py --launcher pytorch
The module torch.distributed.launch is deprecated and going to be removed in future.Migrate to torch.distributed.run
WARNING:torch.distributed.run:--use_env is deprecated and will be removed in future releases.
Please read local_rank from `os.environ('LOCAL_RANK')` instead.
INFO:torch.distributed.launcher.api:Starting elastic_operator with launch configs:
entrypoint : tools/train.py
min_nodes : 1
max_nodes : 1
nproc_per_node : 1
run_id : none
rdzv_backend : static
rdzv_endpoint : 127.0.0.1:29995
rdzv_configs : {'rank': 0, 'timeout': 900}
max_restarts : 3
monitor_interval : 5
log_dir : None
metrics_cfg : {}
INFO:torch.distributed.elastic.agent.server.local_elastic_agent:log directory set to: /tmp/torchelastic_5474cwxi/none_43hbxbvu
INFO:torch.distributed.elastic.agent.server.api:[default] starting workers for entrypoint: python
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous'ing worker group
/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/torch/distributed/elastic/utils/store.py:53: FutureWarning: This is an experimental API and will be changed in future.
"This is an experimental API and will be changed in future.", FutureWarning
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result:
restart_count=0
master_addr=127.0.0.1
master_port=29995
group_rank=0
group_world_size=1
local_ranks=[0]
role_ranks=[0]
global_ranks=[0]
role_world_sizes=[1]
global_world_sizes=[1]
INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group
INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_5474cwxi/none_43hbxbvu/attempt_0/0/error.json
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda-11.3'
Traceback (most recent call last):
File "tools/train.py", line 198, in <module>
main()
File "tools/train.py", line 130, in main
init_dist(args.launcher, **cfg.dist_params)
File "/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/mmcv/runner/dist_utils.py", line 18, in init_dist
_init_dist_pytorch(backend, **kwargs)
File "/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/mmcv/runner/dist_utils.py", line 31, in _init_dist_pytorch
torch.cuda.set_device(rank % num_gpus)
ZeroDivisionError: integer division or modulo by zero
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 5867) of binary: /home/vefak/Documents/anaconda3/envs/teacher/bin/python
ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed
INFO:torch.distributed.elastic.agent.server.api:[default] Worker group FAILED. 3/3 attempts left; will restart worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Stopping worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous'ing worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result:
restart_count=1
master_addr=127.0.0.1
master_port=29995
group_rank=0
group_world_size=1
local_ranks=[0]
role_ranks=[0]
global_ranks=[0]
role_world_sizes=[1]
global_world_sizes=[1]
INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group
INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_5474cwxi/none_43hbxbvu/attempt_1/0/error.json
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda-11.3'
Traceback (most recent call last):
File "tools/train.py", line 198, in <module>
main()
File "tools/train.py", line 130, in main
init_dist(args.launcher, **cfg.dist_params)
File "/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/mmcv/runner/dist_utils.py", line 18, in init_dist
_init_dist_pytorch(backend, **kwargs)
File "/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/mmcv/runner/dist_utils.py", line 31, in _init_dist_pytorch
torch.cuda.set_device(rank % num_gpus)
ZeroDivisionError: integer division or modulo by zero
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 5879) of binary: /home/vefak/Documents/anaconda3/envs/teacher/bin/python
ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed
INFO:torch.distributed.elastic.agent.server.api:[default] Worker group FAILED. 2/3 attempts left; will restart worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Stopping worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous'ing worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result:
restart_count=2
master_addr=127.0.0.1
master_port=29995
group_rank=0
group_world_size=1
local_ranks=[0]
role_ranks=[0]
global_ranks=[0]
role_world_sizes=[1]
global_world_sizes=[1]
INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group
INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_5474cwxi/none_43hbxbvu/attempt_2/0/error.json
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda-11.3'
Traceback (most recent call last):
File "tools/train.py", line 198, in <module>
main()
File "tools/train.py", line 130, in main
init_dist(args.launcher, **cfg.dist_params)
File "/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/mmcv/runner/dist_utils.py", line 18, in init_dist
_init_dist_pytorch(backend, **kwargs)
File "/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/mmcv/runner/dist_utils.py", line 31, in _init_dist_pytorch
torch.cuda.set_device(rank % num_gpus)
ZeroDivisionError: integer division or modulo by zero
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 5891) of binary: /home/vefak/Documents/anaconda3/envs/teacher/bin/python
ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed
INFO:torch.distributed.elastic.agent.server.api:[default] Worker group FAILED. 1/3 attempts left; will restart worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Stopping worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous'ing worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result:
restart_count=3
master_addr=127.0.0.1
master_port=29995
group_rank=0
group_world_size=1
local_ranks=[0]
role_ranks=[0]
global_ranks=[0]
role_world_sizes=[1]
global_world_sizes=[1]
INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group
INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_5474cwxi/none_43hbxbvu/attempt_3/0/error.json
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda-11.3'
Traceback (most recent call last):
File "tools/train.py", line 198, in <module>
main()
File "tools/train.py", line 130, in main
init_dist(args.launcher, **cfg.dist_params)
File "/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/mmcv/runner/dist_utils.py", line 18, in init_dist
_init_dist_pytorch(backend, **kwargs)
File "/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/mmcv/runner/dist_utils.py", line 31, in _init_dist_pytorch
torch.cuda.set_device(rank % num_gpus)
ZeroDivisionError: integer division or modulo by zero
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 5904) of binary: /home/vefak/Documents/anaconda3/envs/teacher/bin/python
ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed
INFO:torch.distributed.elastic.agent.server.api:Local worker group finished (FAILED). Waiting 300 seconds for other agents to finish
/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/torch/distributed/elastic/utils/store.py:71: FutureWarning: This is an experimental API and will be changed in future.
"This is an experimental API and will be changed in future.", FutureWarning
INFO:torch.distributed.elastic.agent.server.api:Done waiting for other agents. Elapsed: 0.00039005279541015625 seconds
{"name": "torchelastic.worker.status.FAILED", "source": "WORKER", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": 0, "group_rank": 0, "worker_id": "5904", "role": "default", "hostname": "vefak", "state": "FAILED", "total_run_time": 20, "rdzv_backend": "static", "raw_error": "{\"message\": \"<NONE>\"}", "metadata": "{\"group_world_size\": 1, \"entry_point\": \"python\", \"local_rank\": [0], \"role_rank\": [0], \"role_world_size\": [1]}", "agent_restarts": 3}}
{"name": "torchelastic.worker.status.SUCCEEDED", "source": "AGENT", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": null, "group_rank": 0, "worker_id": null, "role": "default", "hostname": "vefak", "state": "SUCCEEDED", "total_run_time": 20, "rdzv_backend": "static", "raw_error": null, "metadata": "{\"group_world_size\": 1, \"entry_point\": \"python\"}", "agent_restarts": 3}}
/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py:354: UserWarning:
**********************************************************************
CHILD PROCESS FAILED WITH NO ERROR_FILE
**********************************************************************
CHILD PROCESS FAILED WITH NO ERROR_FILE
Child process 5904 (local_rank 0) FAILED (exitcode 1)
Error msg: Process failed with exitcode 1
Without writing an error file to <N/A>.
While this DOES NOT affect the correctness of your application,
no trace information about the error will be available for inspection.
Consider decorating your top level entrypoint function with
torch.distributed.elastic.multiprocessing.errors.record. Example:
from torch.distributed.elastic.multiprocessing.errors import record
@record
def trainer_main(args):
# do train
**********************************************************************
warnings.warn(_no_error_file_warning_msg(rank, failure))
Traceback (most recent call last):
File "/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/torch/distributed/launch.py", line 173, in <module>
main()
File "/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/torch/distributed/launch.py", line 169, in main
run(args)
File "/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/torch/distributed/run.py", line 624, in run
)(*cmd_args)
File "/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/torch/distributed/launcher/api.py", line 116, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 348, in wrapper
return f(*args, **kwargs)
File "/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/torch/distributed/launcher/api.py", line 247, in launch_agent
failures=result.failures,
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
***************************************
tools/train.py FAILED
=======================================
Root Cause:
[0]:
time: 2022-02-14_21:24:59
rank: 0 (local_rank: 0)
exitcode: 1 (pid: 5904)
error_file: <N/A>
msg: "Process failed with exitcode 1"
=======================================
Other Failures:
<NO_OTHER_FAILURES>
***************************************
The second one, I got different error but I can count it as progress. This time models and annotations are loaded. I saw that it is about using non-distributed mode. Mentioned in #37. What should I do? I cannot use normal mode and non-distributed mode raised different error. The second command and its output
python tools/train.py configs/baseline/faster_rcnn_r50_caffe_fpn_coco_full_720k.py --gpus 1
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda-11.3'
2022-02-14 00:07:51,433 - mmdet.ssod - INFO - [<StreamHandler <stderr> (INFO)>, <FileHandler /home/vefak/Documents/SoftTeacher/work_dirs/faster_rcnn_r50_caffe_fpn_coco_full_720k/20220214_000751.log (INFO)>]
2022-02-14 00:07:51,434 - mmdet.ssod - INFO - Environment info:
------------------------------------------------------------
sys.platform: linux
Python: 3.6.13 |Anaconda, Inc.| (default, Jun 4 2021, 14:25:59) [GCC 7.5.0]
CUDA available: False
GCC: gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
PyTorch: 1.9.0
PyTorch compiling details: PyTorch built with:
- GCC 7.3
- C++ Version: 201402
- Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v2.1.2 (Git Hash 98be7e8afa711dc9b66c8ff3504129cb82013cdb)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- NNPACK is enabled
- CPU capability usage: AVX2
- Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.9.0, USE_CUDA=0, USE_CUDNN=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON,
TorchVision: 0.10.0
OpenCV: 4.5.5
MMCV: 1.3.16
MMCV Compiler: GCC 9.3
MMCV CUDA Compiler: not available
MMDetection: 2.16.0+bef9a25
------------------------------------------------------------
2022-02-14 00:07:51,882 - mmdet.ssod - INFO - Distributed training: False
2022-02-14 00:07:52,332 - mmdet.ssod - INFO - Config:
model = dict(
type='FasterRCNN',
backbone=dict(
type='ResNet',
depth=50,
num_stages=4,
out_indices=(0, 1, 2, 3),
frozen_stages=1,
norm_cfg=dict(type='BN', requires_grad=False),
norm_eval=True,
style='caffe',
init_cfg=dict(
type='Pretrained',
checkpoint='open-mmlab://detectron2/resnet50_caffe')),
neck=dict(
type='FPN',
in_channels=[256, 512, 1024, 2048],
out_channels=256,
num_outs=5),
rpn_head=dict(
type='RPNHead',
in_channels=256,
feat_channels=256,
anchor_generator=dict(
type='AnchorGenerator',
scales=[8],
ratios=[0.5, 1.0, 2.0],
strides=[4, 8, 16, 32, 64]),
bbox_coder=dict(
type='DeltaXYWHBBoxCoder',
target_means=[0.0, 0.0, 0.0, 0.0],
target_stds=[1.0, 1.0, 1.0, 1.0]),
loss_cls=dict(
type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0),
loss_bbox=dict(type='L1Loss', loss_weight=1.0)),
roi_head=dict(
type='StandardRoIHead',
bbox_roi_extractor=dict(
type='SingleRoIExtractor',
roi_layer=dict(type='RoIAlign', output_size=7, sampling_ratio=0),
out_channels=256,
featmap_strides=[4, 8, 16, 32]),
bbox_head=dict(
type='Shared2FCBBoxHead',
in_channels=256,
fc_out_channels=1024,
roi_feat_size=7,
num_classes=80,
bbox_coder=dict(
type='DeltaXYWHBBoxCoder',
target_means=[0.0, 0.0, 0.0, 0.0],
target_stds=[0.1, 0.1, 0.2, 0.2]),
reg_class_agnostic=False,
loss_cls=dict(
type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0),
loss_bbox=dict(type='L1Loss', loss_weight=1.0))),
train_cfg=dict(
rpn=dict(
assigner=dict(
type='MaxIoUAssigner',
pos_iou_thr=0.7,
neg_iou_thr=0.3,
min_pos_iou=0.3,
match_low_quality=True,
ignore_iof_thr=-1),
sampler=dict(
type='RandomSampler',
num=256,
pos_fraction=0.5,
neg_pos_ub=-1,
add_gt_as_proposals=False),
allowed_border=-1,
pos_weight=-1,
debug=False),
rpn_proposal=dict(
nms_pre=2000,
max_per_img=1000,
nms=dict(type='nms', iou_threshold=0.7),
min_bbox_size=0),
rcnn=dict(
assigner=dict(
type='MaxIoUAssigner',
pos_iou_thr=0.5,
neg_iou_thr=0.5,
min_pos_iou=0.5,
match_low_quality=False,
ignore_iof_thr=-1),
sampler=dict(
type='RandomSampler',
num=512,
pos_fraction=0.25,
neg_pos_ub=-1,
add_gt_as_proposals=True),
pos_weight=-1,
debug=False)),
test_cfg=dict(
rpn=dict(
nms_pre=1000,
max_per_img=1000,
nms=dict(type='nms', iou_threshold=0.7),
min_bbox_size=0),
rcnn=dict(
score_thr=0.05,
nms=dict(type='nms', iou_threshold=0.5),
max_per_img=100)))
dataset_type = 'CocoDataset'
data_root = 'data/coco/'
img_norm_cfg = dict(
mean=[103.53, 116.28, 123.675], std=[1.0, 1.0, 1.0], to_rgb=False)
train_pipeline = [
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations', with_bbox=True),
dict(
type='Sequential',
transforms=[
dict(
type='RandResize',
img_scale=[(1333, 400), (1333, 1200)],
multiscale_mode='range',
keep_ratio=True),
dict(type='RandFlip', flip_ratio=0.5),
dict(
type='OneOf',
transforms=[
dict(type='Identity'),
dict(type='AutoContrast'),
dict(type='RandEqualize'),
dict(type='RandSolarize'),
dict(type='RandColor'),
dict(type='RandContrast'),
dict(type='RandBrightness'),
dict(type='RandSharpness'),
dict(type='RandPosterize')
])
]),
dict(type='Pad', size_divisor=32),
dict(
type='Normalize',
mean=[103.53, 116.28, 123.675],
std=[1.0, 1.0, 1.0],
to_rgb=False),
dict(type='ExtraAttrs', tag='sup'),
dict(type='DefaultFormatBundle'),
dict(
type='Collect',
keys=['img', 'gt_bboxes', 'gt_labels'],
meta_keys=('filename', 'ori_shape', 'img_shape', 'img_norm_cfg',
'pad_shape', 'scale_factor', 'tag'))
]
test_pipeline = [
dict(type='LoadImageFromFile'),
dict(
type='MultiScaleFlipAug',
img_scale=(1333, 800),
flip=False,
transforms=[
dict(type='Resize', keep_ratio=True),
dict(type='RandomFlip'),
dict(
type='Normalize',
mean=[103.53, 116.28, 123.675],
std=[1.0, 1.0, 1.0],
to_rgb=False),
dict(type='Pad', size_divisor=32),
dict(type='ImageToTensor', keys=['img']),
dict(type='Collect', keys=['img'])
])
]
data = dict(
samples_per_gpu=2,
workers_per_gpu=2,
train=dict(
type='CocoDataset',
ann_file='data/coco/annotations/instances_train2017.json',
img_prefix='data/coco/train2017/',
pipeline=[
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations', with_bbox=True),
dict(
type='Sequential',
transforms=[
dict(
type='RandResize',
img_scale=[(1333, 400), (1333, 1200)],
multiscale_mode='range',
keep_ratio=True),
dict(type='RandFlip', flip_ratio=0.5),
dict(
type='OneOf',
transforms=[
dict(type='Identity'),
dict(type='AutoContrast'),
dict(type='RandEqualize'),
dict(type='RandSolarize'),
dict(type='RandColor'),
dict(type='RandContrast'),
dict(type='RandBrightness'),
dict(type='RandSharpness'),
dict(type='RandPosterize')
])
]),
dict(type='Pad', size_divisor=32),
dict(
type='Normalize',
mean=[103.53, 116.28, 123.675],
std=[1.0, 1.0, 1.0],
to_rgb=False),
dict(type='ExtraAttrs', tag='sup'),
dict(type='DefaultFormatBundle'),
dict(
type='Collect',
keys=['img', 'gt_bboxes', 'gt_labels'],
meta_keys=('filename', 'ori_shape', 'img_shape',
'img_norm_cfg', 'pad_shape', 'scale_factor', 'tag'))
]),
val=dict(
type='CocoDataset',
ann_file='data/coco/annotations/instances_val2017.json',
img_prefix='data/coco/val2017/',
pipeline=[
dict(type='LoadImageFromFile'),
dict(
type='MultiScaleFlipAug',
img_scale=(1333, 800),
flip=False,
transforms=[
dict(type='Resize', keep_ratio=True),
dict(type='RandomFlip'),
dict(
type='Normalize',
mean=[103.53, 116.28, 123.675],
std=[1.0, 1.0, 1.0],
to_rgb=False),
dict(type='Pad', size_divisor=32),
dict(type='ImageToTensor', keys=['img']),
dict(type='Collect', keys=['img'])
])
]),
test=dict(
type='CocoDataset',
ann_file='data/coco/annotations/instances_val2017.json',
img_prefix='data/coco/val2017/',
pipeline=[
dict(type='LoadImageFromFile'),
dict(
type='MultiScaleFlipAug',
img_scale=(1333, 800),
flip=False,
transforms=[
dict(type='Resize', keep_ratio=True),
dict(type='RandomFlip'),
dict(
type='Normalize',
mean=[103.53, 116.28, 123.675],
std=[1.0, 1.0, 1.0],
to_rgb=False),
dict(type='Pad', size_divisor=32),
dict(type='ImageToTensor', keys=['img']),
dict(type='Collect', keys=['img'])
])
]))
evaluation = dict(interval=4000, metric='bbox')
optimizer = dict(type='SGD', lr=0.02, momentum=0.9, weight_decay=0.0001)
optimizer_config = dict(grad_clip=None)
lr_config = dict(
policy='step',
warmup='linear',
warmup_iters=500,
warmup_ratio=0.001,
step=[480000, 640000])
runner = dict(type='IterBasedRunner', max_iters=720000)
checkpoint_config = dict(interval=4000, by_epoch=False, max_keep_ckpts=10)
log_config = dict(
interval=50,
hooks=[
dict(type='TextLoggerHook', by_epoch=False),
dict(
type='WandbLoggerHook',
init_kwargs=dict(
project='pre_release',
name='faster_rcnn_r50_caffe_fpn_coco_full_720k',
config=dict(
work_dirs=
'./work_dirs/faster_rcnn_r50_caffe_fpn_coco_full_720k',
total_step=720000)),
by_epoch=False)
])
custom_hooks = [dict(type='NumClassCheckHook')]
dist_params = dict(backend='nccl')
log_level = 'INFO'
load_from = None
resume_from = None
workflow = [('train', 1)]
mmdet_base = '../../thirdparty/mmdetection/configs/_base_'
fp16 = dict(loss_scale='dynamic')
work_dir = './work_dirs/faster_rcnn_r50_caffe_fpn_coco_full_720k'
cfg_name = 'faster_rcnn_r50_caffe_fpn_coco_full_720k'
gpu_ids = range(0, 1)
/home/vefak/Documents/SoftTeacher/thirdparty/mmdetection/mmdet/core/anchor/builder.py:17: UserWarning: ``build_anchor_generator`` would be deprecated soon, please use ``build_prior_generator``
'``build_anchor_generator`` would be deprecated soon, please use '
2022-02-14 00:07:52,754 - mmdet.ssod - INFO - initialize ResNet with init_cfg {'type': 'Pretrained', 'checkpoint': 'open-mmlab://detectron2/resnet50_caffe'}
2022-02-14 00:07:52,755 - mmcv - INFO - load model from: open-mmlab://detectron2/resnet50_caffe
2022-02-14 00:07:52,755 - mmcv - INFO - Use load_from_openmmlab loader
2022-02-14 00:07:52,840 - mmcv - WARNING - The model and loaded state dict do not match exactly
unexpected key in source state_dict: conv1.bias
2022-02-14 00:07:52,854 - mmdet.ssod - INFO - initialize FPN with init_cfg {'type': 'Xavier', 'layer': 'Conv2d', 'distribution': 'uniform'}
2022-02-14 00:07:52,880 - mmdet.ssod - INFO - initialize RPNHead with init_cfg {'type': 'Normal', 'layer': 'Conv2d', 'std': 0.01}
2022-02-14 00:07:52,884 - mmdet.ssod - INFO - initialize Shared2FCBBoxHead with init_cfg [{'type': 'Normal', 'std': 0.01, 'override': {'name': 'fc_cls'}}, {'type': 'Normal', 'std': 0.001, 'override': {'name': 'fc_reg'}}, {'type': 'Xavier', 'layer': 'Linear', 'override': [{'name': 'shared_fcs'}, {'name': 'cls_fcs'}, {'name': 'reg_fcs'}]}]
loading annotations into memory...
Done (t=14.22s)
creating index...
index created!
Traceback (most recent call last):
File "tools/train.py", line 198, in <module>
main()
File "tools/train.py", line 193, in main
meta=meta,
File "/home/vefak/Documents/SoftTeacher/ssod/apis/train.py", line 81, in train_detector
for ds in dataset
File "/home/vefak/Documents/SoftTeacher/ssod/apis/train.py", line 81, in <listcomp>
for ds in dataset
File "/home/vefak/Documents/SoftTeacher/ssod/datasets/builder.py", line 69, in build_dataloader
if shuffle
File "/home/vefak/Documents/SoftTeacher/ssod/datasets/builder.py", line 40, in build_sampler
return build_from_cfg(cfg, SAMPLERS, default_args)
File "/home/vefak/Documents/anaconda3/envs/teacher/lib/python3.6/site-packages/mmcv/utils/registry.py", line 45, in build_from_cfg
f'{obj_type} is not in the {registry.name} registry')
KeyError: 'Sampler is not in the sampler registry'
Hi again, Is there any updates for non distrubed version?
The problem is about incompability between RTX3050, pytorch and mmcv-full RTX3050 doesnot support CUDA 10.x version. It should be specific configuration between them.
Currently I installed CUDA 11.1 Pytorch 1.9.0 mmcv 1.3.9
Hi there, I'm running into this issue if my annotation.json file is huge(7Gb).
I am able to train my model on multiple GPUs if I use a smaller slice of the full dataset using the command:
!CUDA_VISIBLE_DEVICES=0,1,2,3 PORT=29501 ./tools/dist_train.sh ./configs/motsynth/qdtrack_frcnn_r50_fpn_4e_motsynth.py 4 --work-dir work_dirs/MOTSynth/virgin --cfg-options 'optimizer.lr=0.01' 'data.train.ann_file=data/MOTSynth/annotations/test_cocoformat.json'
,
and I can train it on the full dataset when I use a single GPU using the following command:
!python ./tools/train.py ./configs/motsynth/qdtrack_frcnn_r50_fpn_4e_motsynth.py --cfg-options work_dir="work_dirs/MOTSynth/virgin" optimizer.lr=0.0025
.
My environment has the following packages:
I have the same issue, what's going on this problem?
The module torch.distributed.launch is deprecated and going to be removed in future.Migrate to torch.distributed.run WARNING:torch.distributed.run:--use_env is deprecated and will be removed in future releases. Please read local_rank from
os.environ('LOCAL_RANK')` instead. INFO:torch.distributed.launcher.api:Starting elastic_operator with launch configs: entrypoint : tools/train.py min_nodes : 1 max_nodes : 1 nproc_per_node : 1 run_id : none rdzv_backend : static rdzv_endpoint : 127.0.0.1:29500 rdzv_configs : {'rank': 0, 'timeout': 900} max_restarts : 3 monitor_interval : 5 log_dir : None metrics_cfg : {}INFO:torch.distributed.elastic.agent.server.local_elastic_agent:log directory set to: /tmp/torchelastic_mnn5x9jo/none_j57hpvun INFO:torch.distributed.elastic.agent.server.api:[default] starting workers for entrypoint: python3 INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous'ing worker group /home/vefak/Documents/anaconda3/envs/torch/lib/python3.6/site-packages/torch/distributed/elastic/utils/store.py:53: FutureWarning: This is an experimental API and will be changed in future. "This is an experimental API and will be changed in future.", FutureWarning INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result: restart_count=0 master_addr=127.0.0.1 master_port=29500 group_rank=0 group_world_size=1 local_ranks=[0] role_ranks=[0] global_ranks=[0] role_world_sizes=[1] global_world_sizes=[1]
INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_mnn5x9jo/none_j57hpvun/attempt_0/0/error.json ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -11) local_rank: 0 (pid: 14927) of binary: /home/vefak/Documents/anaconda3/envs/torch/bin/python3 ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed INFO:torch.distributed.elastic.agent.server.api:[default] Worker group FAILED. 3/3 attempts left; will restart worker group INFO:torch.distributed.elastic.agent.server.api:[default] Stopping worker group INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous'ing worker group INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result: restart_count=1 master_addr=127.0.0.1 master_port=29500 group_rank=0 group_world_size=1 local_ranks=[0] role_ranks=[0] global_ranks=[0] role_world_sizes=[1] global_world_sizes=[1]
INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_mnn5x9jo/none_j57hpvun/attempt_1/0/error.json ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -11) local_rank: 0 (pid: 14955) of binary: /home/vefak/Documents/anaconda3/envs/torch/bin/python3 ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed INFO:torch.distributed.elastic.agent.server.api:[default] Worker group FAILED. 2/3 attempts left; will restart worker group INFO:torch.distributed.elastic.agent.server.api:[default] Stopping worker group INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous'ing worker group INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result: restart_count=2 master_addr=127.0.0.1 master_port=29500 group_rank=0 group_world_size=1 local_ranks=[0] role_ranks=[0] global_ranks=[0] role_world_sizes=[1] global_world_sizes=[1]
INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_mnn5x9jo/none_j57hpvun/attempt_2/0/error.json ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -11) local_rank: 0 (pid: 15007) of binary: /home/vefak/Documents/anaconda3/envs/torch/bin/python3 ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed INFO:torch.distributed.elastic.agent.server.api:[default] Worker group FAILED. 1/3 attempts left; will restart worker group INFO:torch.distributed.elastic.agent.server.api:[default] Stopping worker group INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous'ing worker group INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result: restart_count=3 master_addr=127.0.0.1 master_port=29500 group_rank=0 group_world_size=1 local_ranks=[0] role_ranks=[0] global_ranks=[0] role_world_sizes=[1] global_world_sizes=[1]
INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_mnn5x9jo/none_j57hpvun/attempt_3/0/error.json ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -11) local_rank: 0 (pid: 15048) of binary: /home/vefak/Documents/anaconda3/envs/torch/bin/python3 ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed INFO:torch.distributed.elastic.agent.server.api:Local worker group finished (FAILED). Waiting 300 seconds for other agents to finish /home/vefak/Documents/anaconda3/envs/torch/lib/python3.6/site-packages/torch/distributed/elastic/utils/store.py:71: FutureWarning: This is an experimental API and will be changed in future. "This is an experimental API and will be changed in future.", FutureWarning INFO:torch.distributed.elastic.agent.server.api:Done waiting for other agents. Elapsed: 0.0004889965057373047 seconds {"name": "torchelastic.worker.status.FAILED", "source": "WORKER", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": 0, "group_rank": 0, "worker_id": "15048", "role": "default", "hostname": "vefak", "state": "FAILED", "total_run_time": 25, "rdzv_backend": "static", "raw_error": "{\"message\": \"\"}", "metadata": "{\"group_world_size\": 1, \"entry_point\": \"python3\", \"local_rank\": [0], \"role_rank\": [0], \"role_world_size\": [1]}", "agent_restarts": 3}}
{"name": "torchelastic.worker.status.SUCCEEDED", "source": "AGENT", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": null, "group_rank": 0, "worker_id": null, "role": "default", "hostname": "vefak", "state": "SUCCEEDED", "total_run_time": 25, "rdzv_backend": "static", "raw_error": null, "metadata": "{\"group_world_size\": 1, \"entry_point\": \"python3\"}", "agent_restarts": 3}}
/home/vefak/Documents/anaconda3/envs/torch/lib/python3.6/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py:354: UserWarning:
CHILD PROCESS FAILED WITH NO ERROR_FILE Child process 15048 (local_rank 0) FAILED (exitcode -11) Error msg: Signal 11 (SIGSEGV) received by PID 15048 Without writing an error file to <N/A>. While this DOES NOT affect the correctness of your application, no trace information about the error will be available for inspection. Consider decorating your top level entrypoint function with torch.distributed.elastic.multiprocessing.errors.record. Example:
from torch.distributed.elastic.multiprocessing.errors import record
@record def trainer_main(args):
do train
warnings.warn(_no_error_file_warning_msg(rank, failure)) Traceback (most recent call last): File "/home/vefak/Documents/anaconda3/envs/torch/lib/python3.6/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/home/vefak/Documents/anaconda3/envs/torch/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/vefak/Documents/anaconda3/envs/torch/lib/python3.6/site-packages/torch/distributed/launch.py", line 173, in
main()
File "/home/vefak/Documents/anaconda3/envs/torch/lib/python3.6/site-packages/torch/distributed/launch.py", line 169, in main
run(args)
File "/home/vefak/Documents/anaconda3/envs/torch/lib/python3.6/site-packages/torch/distributed/run.py", line 624, in run
)(cmd_args)
File "/home/vefak/Documents/anaconda3/envs/torch/lib/python3.6/site-packages/torch/distributed/launcher/api.py", line 116, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/vefak/Documents/anaconda3/envs/torch/lib/python3.6/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 348, in wrapper
return f(args, **kwargs)
File "/home/vefak/Documents/anaconda3/envs/torch/lib/python3.6/site-packages/torch/distributed/launcher/api.py", line 247, in launch_agent
failures=result.failures,
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
================================================== Root Cause: [0]: time: 2022-02-04_00:19:07 rank: 0 (local_rank: 0) exitcode: -11 (pid: 15048) error_file: <N/A> msg: "Signal 11 (SIGSEGV) received by PID 15048"
Other Failures: