search

chenhang98 / BPR

code for `Look Closer to Segment Better: Boundary Patch Refinement for Instance Segmentation`
Apache License 2.0
173 stars 23 forks source link

Training not distributed #31

Closed Lopside1 closed 2 years ago

Lopside1 commented 2 years ago

Hi I am calling the dist_train but getting errors that seem related to the distributed part of the code. Is there a way I can train without calling the distribute wrapper script as I only have the 1 GPU.

Lopside1 commented 2 years ago

Output from

DATA_ROOT=data/patches bash tools/dist_train.sh configs/bpr/hrnet18s_128.py 1

------------------------------------------------------------
sys.platform: linux
Python: 3.7.13 (default, Mar 29 2022, 02:18:16) [GCC 7.5.0]
CUDA available: True
GPU 0: NVIDIA GeForce GTX 1060
CUDA_HOME: None
GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
PyTorch: 1.6.0
PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v1.5.0 (Git Hash e2ac1fac44c5078ca927cb9b90e1b3066a0b2ed0)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 10.1
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_37,code=compute_37
  - CuDNN 7.6.3
  - Magma 2.5.2
  - Build settings: BLAS=MKL, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_VULKAN_WRAPPER -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_STATIC_DISPATCH=OFF,

TorchVision: 0.7.0
OpenCV: 4.5.5
MMCV: 1.1.6
MMCV Compiler: GCC 7.3
MMCV CUDA Compiler: 10.1
MMSegmentation: 0.7.0+a615828

...

  (decode_head): FCNHead(
    input_transform=resize_concat, ignore_index=255, align_corners=False
    (loss_decode): CrossEntropyLoss()
    (conv_seg): Conv2d(270, 2, kernel_size=(1, 1), stride=(1, 1))
    (convs): Sequential(
      (0): ConvModule(
        (conv): Conv2d(270, 270, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn): SyncBatchNorm(270, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (activate): ReLU(inplace=True)
      )
    )
  )
)
2022-05-12 02:23:26,401 - mmseg - INFO - Loaded 27162 images
LAPTOP-01C1HLPT:16453:16453 [0] NCCL INFO Bootstrap : Using [0]eth0:172.19.69.7<0>
LAPTOP-01C1HLPT:16453:16453 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).

LAPTOP-01C1HLPT:16453:16453 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
LAPTOP-01C1HLPT:16453:16453 [0] NCCL INFO NET/Socket : Using [0]eth0:172.19.69.7<0>
NCCL version 2.4.8+cuda10.1

LAPTOP-01C1HLPT:16453:16472 [0] misc/topo.cc:22 NCCL WARN Could not find real path of /sys/class/pci_bus/0000:01/../../0000:01:00.0
LAPTOP-01C1HLPT:16453:16472 [0] NCCL INFO init.cc:876 -> 2
LAPTOP-01C1HLPT:16453:16472 [0] NCCL INFO init.cc:909 -> 2
LAPTOP-01C1HLPT:16453:16472 [0] NCCL INFO init.cc:947 -> 2
LAPTOP-01C1HLPT:16453:16472 [0] NCCL INFO misc/group.cc:69 -> 2 [Async thread]
Traceback (most recent call last):
  File "tools/train.py", line 165, in <module>
    main()
  File "tools/train.py", line 161, in main
    meta=meta)
  File "/home/lryan/BPR/mmseg/apis/train.py", line 66, in train_segmentor
    find_unused_parameters=find_unused_parameters)
  File "/home/lryan/miniconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 333, in __init__
    self.broadcast_bucket_size)
  File "/home/lryan/miniconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 549, in _distributed_broadcast_coalesced
    dist._broadcast_coalesced(self.process_group, tensors, buffer_size)
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1595629403081/work/torch/lib/c10d/ProcessGroupNCCL.cpp:518, unhandled system error, NCCL version 2.4.8
Traceback (most recent call last):
  File "/home/lryan/miniconda3/envs/open-mmlab/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/lryan/miniconda3/envs/open-mmlab/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/lryan/miniconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/distributed/launch.py", line 261, in <module>
    main()
  File "/home/lryan/miniconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/distributed/launch.py", line 257, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/home/lryan/miniconda3/envs/open-mmlab/bin/python', '-u', 'tools/train.py', '--local_rank=0', 'configs/bpr/hrnet18s_128.py', '--launcher', 'pytorch']' returned non-zero exit status 1.
chenhang98 commented 2 years ago

Yes, you can train on a single GPU. Just replace bash tools/dist_train.sh with python tools/train.py and delete the 1.

Lopside1 commented 2 years ago

Hi, thank you for getting back again.

I did try this also but get the error that the EncoderDecoderRefine is not in the Segmenter registry. I have managed to produce this on 2 separate machines with different installs and tried to debug into it to find out the issue. 

DATA_ROOT=data/patches/ python tools/train.py configs/bpr/hrnet18s_128.py 
2022-05-12 18:55:42,780 - mmseg - INFO - Environment info:
------------------------------------------------------------
sys.platform: linux
Python: 3.7.12 | packaged by conda-forge | (default, Oct 26 2021, 06:08:21) [GCC 9.4.0]
CUDA available: True
GPU 0: Tesla K80
CUDA_HOME: /usr/local/cuda
NVCC: Build cuda_11.0_bu.TC445_37.28845127_0
GCC: gcc (conda-forge gcc 11.2.0-16) 11.2.0
PyTorch: 1.7.0
PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v1.6.0 (Git Hash 5ef631a030a6f73131c77892041042805a06064f)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 11.0
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_37,code=compute_37
  - CuDNN 8.0.3
  - Magma 2.5.2
  - Build settings: BLAS=MKL, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_VULKAN_WRAPPER -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, 

TorchVision: 0.8.1
OpenCV: 4.5.5
MMCV: 1.2.0
MMCV Compiler: GCC 7.3
MMCV CUDA Compiler: not available
MMSegmentation: 0.9.0+a615828
------------------------------------------------------------

2022-05-12 18:55:42,780 - mmseg - INFO - Distributed training: False
2022-05-12 18:55:43,074 - mmseg - INFO - Config:

...

work_dir = './work_dirs/hrnet18s_128'
gpu_ids = range(0, 1)

Traceback (most recent call last):
  File "tools/train.py", line 161, in <module>
    main()
  File "tools/train.py", line 131, in main
    cfg.model, train_cfg=cfg.train_cfg, test_cfg=cfg.test_cfg)
  File "/opt/conda/envs/open-mmlab/lib/python3.7/site-packages/mmseg/models/builder.py", line 56, in build_segmentor
    return build(cfg, SEGMENTORS, dict(train_cfg=train_cfg, test_cfg=test_cfg))
  File "/opt/conda/envs/open-mmlab/lib/python3.7/site-packages/mmseg/models/builder.py", line 31, in build
    return build_from_cfg(cfg, registry, default_args)
  File "/opt/conda/envs/open-mmlab/lib/python3.7/site-packages/mmcv/utils/registry.py", line 164, in build_from_cfg
    f'{obj_type} is not in the {registry.name} registry')
KeyError: 'EncoderDecoderRefine is not in the segmentor registry'
chenhang98 commented 2 years ago

It seems that you have the wrong mmsegmentation installed. You can install our repo from source.

chenhang98 commented 2 years ago

i.e. use pip install -e .

Lopside1 commented 2 years ago

thanks I will look into this

Lopside1 commented 2 years ago

Hi, I now have it running and training successfully. There was one last issue of changing SyncBN to BN for non-distributed training.

Thank you for all your help and this repo :-)