Closed korneliaWatson closed 1 year ago
Hi, thanks for using MMPose. You may need to install mmpose before training via pip install -e /home/user/project/mmpose
Right, thank you. After running pip install -e /home/user/project/mmpose
I now get another error.
/opt/conda/lib/python3.10/site-packages/mmcv/cnn/bricks/transformer.py:33: UserWarning: Fail to import ``MultiScaleDeformableAttention`` from ``mmcv.ops.multi_scale_deform_attn``, You should install ``mmcv`` rather than ``mmcv-lite`` if you need this module.
warnings.warn('Fail to import ``MultiScaleDeformableAttention`` from '
07/06 04:13:29 - mmengine - INFO - Distributed training is not used, all SyncBatchNorm (SyncBN) layers in the model will be automatically reverted to BatchNormXd layers if they are used.
Traceback (most recent call last):
File "/home/user/project/mmpose/tools/train.py", line 160, in <module>
main()
File "/home/user/project/mmpose/tools/train.py", line 153, in main
runner = Runner.from_cfg(cfg)
File "/opt/conda/lib/python3.10/site-packages/mmengine/runner/runner.py", line 443, in from_cfg
runner = cls(
File "/opt/conda/lib/python3.10/site-packages/mmengine/runner/runner.py", line 412, in __init__
self.model = self.wrap_model(
File "/opt/conda/lib/python3.10/site-packages/mmengine/runner/runner.py", line 862, in wrap_model
model = revert_sync_batchnorm(model)
File "/opt/conda/lib/python3.10/site-packages/mmengine/model/utils.py", line 174, in revert_sync_batchnorm
from mmcv.ops import SyncBatchNorm
File "/opt/conda/lib/python3.10/site-packages/mmcv/ops/__init__.py", line 2, in <module>
from .active_rotated_filter import active_rotated_filter
File "/opt/conda/lib/python3.10/site-packages/mmcv/ops/active_rotated_filter.py", line 10, in <module>
ext_module = ext_loader.load_ext(
File "/opt/conda/lib/python3.10/site-packages/mmcv/utils/ext_loader.py", line 13, in load_ext
ext = importlib.import_module('mmcv.' + name)
File "/opt/conda/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ImportError: /opt/conda/lib/python3.10/site-packages/mmcv/_ext.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN2at4_ops6narrow4callERKNS_6TensorElll
I have mmcv
installed, though, version 2.0.1.
According to the mmcv FAQ, there might be mismatch between your pytorch version and that for compiling mmcv. We would suggest reinstalling mmcv or build mmcv from source
Hi, thank you. That solved the problem.
I'm now experiencing a new one... I seem to manage to execute 10 first training iterations but once a checkpoint is saved, the training stops with Killed
output. I looked at training config but I'm not seeing anything that would indicate it's supposed to terminate. Any ideas why that might be happening?
07/11 11:43:15 - mmengine - INFO - Epoch(val) [10][10/10] coco/AP: 1.000000 coco/AP .5: 1.000000 coco/AP .75: 1.000000 coco/AP (M): 1.000000 coco/AP (L): 1.000000 coco/AR: 1.000000 coco/AR .5: 1.000000 coco/AR .75: 1.000000 coco/AR (M): 1.000000 coco/AR (L): 1.000000 data_time: 0.056178 time: 6.020677
07/11 11:43:19 - mmengine - INFO - The best checkpoint with 1.0000 coco/AP at 10 epoch is saved to best_coco_AP_epoch_10.pth.
Killed
In Linux, a process can be terminated when there is not enough memory available. To address this issue, it would be beneficial to monitor the memory usage of the training process and detect any occurrence of OOM (out of memory) problems.
Prerequisite
Environment
/opt/conda/lib/python3.10/site-packages/torch/cuda/init.py:107: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibility was attempted on non supported HW (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:109.) return torch._C._cuda_getDeviceCount() > 0 Traceback (most recent call last): File "", line 1, in
File "/opt/conda/lib/python3.10/site-packages/mmcv/utils/env.py", line 72, in collect_env
from mmcv.ops import get_compiler_version, get_compiling_cuda_version
File "/opt/conda/lib/python3.10/site-packages/mmcv/ops/init.py", line 2, in
from .active_rotated_filter import active_rotated_filter
File "/opt/conda/lib/python3.10/site-packages/mmcv/ops/active_rotated_filter.py", line 10, in
ext_module = ext_loader.load_ext(
File "/opt/conda/lib/python3.10/site-packages/mmcv/utils/ext_loader.py", line 13, in load_ext
ext = importlib.import_module('mmcv.' + name)
File "/opt/conda/lib/python3.10/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ImportError: /opt/conda/lib/python3.10/site-packages/mmcv/_ext.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN2at4_ops6narrow4callERKNS_6TensorElll
mmcv 2.0.1 mmdet 3.1.0 mmengine 0.8.0
Reproduces the problem - code sample
Reproduces the problem - command or script
Reproduces the problem - error message
Additional information
I had the training script working previously. All I did was remove my virtual environment to do a fresh install of dependencies. Since then, I couldn't get the script working and I've been experiencing a range of issues that got me circulating in a ring of madness.
I've had several issues getting dependencies working nicely together. After several trial and errors these are the versions that finally managed to install without producing errors.
Output of nvcc --version
I would appreciate help with getting this issue resolved. Thank you.