运行错误

[lqh964165950]

subprocess.CalledProcessError: Command '['/home/gxu4090x2/.conda/envs/cat/bin/python3.11', '-m', 'torch.distributed.launch', '--nproc_per_node=1', '--master_port=26968', '/home/gxu4090x2/.conda/envs/cat/lib/python3.11/site-packages/mmdet/.mim/tools/train.py', 'configs/dior/catnet_r50_3x_dior.py', '--launcher', 'pytorch']' returned non-zero exit status 1. 请问遇到这个问题该怎么解决？

[yeliudev]

Could you please provide more details about the error log and the environment?

[lqh964165950]

报错信息： Using port 22203 for synchronization. Training command is /home/gxu4090x2/.conda/envs/sod/bin/python3.11 -m torch.distributed.launch --nproc_per_node=1 --master_port=22203 /home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/mmdet/.mim/tools/train.py configs/dior/catnet_r50_3x_dior.py --launcher pytorch. /home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/torch/distributed/launch.py:181: FutureWarning: The module torch.distributed.launch is deprecated and will be removed in future. Use torchrun. Note that --use-env is set by default in torchrun. If your script expects --local-rank argument to be set, please change it to read from os.environ['LOCAL_RANK'] instead. See https://pytorch.org/docs/stable/distributed.html#launch-utility for further instructions

warnings.warn(

A module that was compiled using NumPy 1.x cannot be run in NumPy 2.1.1 as it may crash. To support both 1.x and 2.x versions of NumPy, modules must be compiled with NumPy 2.0. Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to downgrade to 'numpy<2' or try to upgrade the affected module. We expect that some modules will need time to support NumPy 2.

Traceback (most recent call last): File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/mmdet/.mim/tools/train.py", line 10, in from mmengine.runner import Runner File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/mmengine/runner/init.py", line 4, in from .checkpoint import (CheckpointLoader, find_latest_checkpoint, File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/mmengine/runner/checkpoint.py", line 20, in from mmengine.model import BaseTTAModel, is_model_wrapper File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/mmengine/model/init.py", line 6, in from .base_model import BaseDataPreprocessor, BaseModel, ImgDataPreprocessor File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/mmengine/model/base_model/init.py", line 2, in from .base_model import BaseModel File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/mmengine/model/base_model/base_model.py", line 9, in from mmengine.optim import OptimWrapper File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/mmengine/optim/init.py", line 2, in from .optimizer import (OPTIM_WRAPPER_CONSTRUCTORS, OPTIMIZERS, File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/mmengine/optim/optimizer/init.py", line 9, in from .zero_optimizer import ZeroRedundancyOptimizer File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/mmengine/optim/optimizer/zero_optimizer.py", line 11, in from torch.distributed.optim import \ File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/torch/distributed/optim/init.py", line 30, in from .zero_redundancy_optimizer import ZeroRedundancyOptimizer File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/torch/distributed/optim/zero_redundancy_optimizer.py", line 74, in device: torch.device = torch.device("cpu"), /home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/torch/distributed/optim/zero_redundancy_optimizer.py:74: UserWarning: Failed to initialize NumPy: _ARRAY_API not found (Triggered internally at ../torch/csrc/utils/tensor_numpy.cpp:84.) device: torch.device = torch.device("cpu"), Traceback (most recent call last): File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/mmengine/utils/misc.py", line 77, in import_modules_from_strings imported_tmp = import_module(imp) ^^^^^^^^^^^^^^^^^^ File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/importlib/init.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "", line 1206, in _gcd_import File "", line 1178, in _find_and_load File "", line 1149, in _find_and_load_unlocked File "", line 690, in _load_unlocked File "", line 940, in exec_module File "", line 241, in _call_with_frames_removed File "/home/gxu4090x2/桌面/lqh/program/CATNet/models/init.py", line 2, in from .hroie import HierarchicalRoIExtractor File "/home/gxu4090x2/桌面/lqh/program/CATNet/models/hroie.py", line 10, in from mmdet.models import BaseRoIExtractor File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/mmdet/models/init.py", line 2, in from .backbones import * # noqa: F401,F403 ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/mmdet/models/backbones/init.py", line 2, in from .csp_darknet import CSPDarknet File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/mmdet/models/backbones/csp_darknet.py", line 11, in from ..layers import CSPLayer File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/mmdet/models/layers/init.py", line 3, in from .bbox_nms import fast_nms, multiclass_nms File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/mmdet/models/layers/bbox_nms.py", line 5, in from mmcv.ops.nms import batched_nms File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/mmcv/ops/init.py", line 2, in from .active_rotated_filter import active_rotated_filter File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/mmcv/ops/active_rotated_filter.py", line 10, in ext_module = ext_loader.load_ext( ^^^^^^^^^^^^^^^^^^^^ File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/mmcv/utils/ext_loader.py", line 13, in load_ext ext = importlib.import_module('mmcv.' + name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/importlib/init.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ModuleNotFoundError: No module named 'mmcv._ext'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/mmengine/config/config.py", line 182, in fromfile import_modules_from_strings(**cfg_dict['custom_imports']) File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/mmengine/utils/misc.py", line 84, in import_modules_from_strings raise ImportError(f'Failed to import {imp}') ImportError: Failed to import models

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/mmdet/.mim/tools/train.py", line 133, in main() File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/mmdet/.mim/tools/train.py", line 70, in main cfg = Config.fromfile(args.config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/mmengine/config/config.py", line 192, in fromfile raise ImportError(err_msg) from e ImportError: Failed to import custom modules from {'imports': ['models', 'datasets']}, the current sys.path is: /home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/mmdet/.mim/tools /home/gxu4090x2/桌面/lqh/program/CATNet /home/gxu4090x2/.conda/envs/sod/lib/python311.zip /home/gxu4090x2/.conda/envs/sod/lib/python3.11 /home/gxu4090x2/.conda/envs/sod/lib/python3.11/lib-dynload /home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages /tmp/tmpu97o2siy You should set PYTHONPATH to make sys.path include the directory which contains your custom module ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 629591) of binary: /home/gxu4090x2/.conda/envs/sod/bin/python3.11 Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/torch/distributed/launch.py", line 196, in main() File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/torch/distributed/launch.py", line 192, in main launch(args) File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/torch/distributed/launch.py", line 177, in launch run(args) File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/torch/distributed/run.py", line 785, in run elastic_launch( File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 134, in call return launch_agent(self._config, self._entrypoint, list(args)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/mmdet/.mim/tools/train.py FAILED

Failures:

------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2024-09-13_16:10:02 host : gxu4090x2-ubuntu rank : 0 (local_rank: 0) exitcode : 1 (pid: 629591) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ============================================================ Traceback (most recent call last): File "/home/gxu4090x2/.conda/envs/sod/bin/mim", line 8, in sys.exit(cli()) ^^^^^ File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/mim/commands/train.py", line 100, in cli is_success, msg = train( ^^^^^^ File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/mim/commands/train.py", line 261, in train ret = subprocess.check_call( ^^^^^^^^^^^^^^^^^^^^^^ File "/home/gxu4090x2/.conda/envs/sod/lib/python3.11/subprocess.py", line 413, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['/home/gxu4090x2/.conda/envs/sod/bin/python3.11', '-m', 'torch.distributed.launch', '--nproc_per_node=1', '--master_port=22203', '/home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages/mmdet/.mim/tools/train.py', 'configs/dior/catnet_r50_3x_dior.py', '--launcher', 'pytorch']' returned non-zero exit status 1. environment： [environment.txt](https://github.com/user-attachments/files/16990668/environment.txt)

[yeliudev]

Looks like the problem from mmcv side. Please make sure mmcv is correctly installed.

[lqh964165950]

After inputting the command: pip show mmcv,the details are: Name: mmcv Version: 2.0.1 Summary: OpenMMLab Computer Vision Foundation Home-page: https://github.com/open-mmlab/mmcv Author: MMCV Contributors Author-email: openmmlab@gmail.com License: Location: /home/gxu4090x2/.conda/envs/sod/lib/python3.11/site-packages Requires: addict, mmengine, numpy, opencv-python, packaging, Pillow, pyyaml, yapf Required-by: How to know whether mmcv is correctly installed？

[yeliudev]

Your error log says ModuleNotFoundError: No module named 'mmcv._ext', which means the CUDA extensions were not compiled successfully, only the Python part was installed. You may refer to mmcv's repo for details.

[lqh964165950]

What should I do？Since mmcv is installed correctly.

[yeliudev]

Please create an issue in mmcv's repo.