PaddlePaddle / PaddleDetection

Object Detection toolkit based on PaddlePaddle. It supports object detection, instance segmentation, multiple object tracking and real-time multi-person keypoint detection.
Apache License 2.0
12.38k stars 2.84k forks source link

[k200 麒麟并行问题]AttributeError: module 'paddle.fluid.core_noavx' has no attribute 'BKCLParallelContext' #5807

Open 1chang37 opened 2 years ago

1chang37 commented 2 years ago

python3 -m paddle.distributed.launch --log_dir=./ppyolo_dygraph/ --xpus 0,1 tools/train.py -c configs/ppyolo/ppyolo_r50vd_dcn_1x_coco.yml

XPURT /usr/local/lib64/python3.7/site-packages/paddle/fluid/../libs/libxpurt.so loaded /usr/lib/python3.7/site-packages/setuptools/depends.py:2: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp /usr/local/lib64/python3.7/site-packages/paddle/vision/transforms/functional_pil.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead. 'nearest': Image.NEAREST, /usr/local/lib64/python3.7/site-packages/paddle/vision/transforms/functional_pil.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead. 'bilinear': Image.BILINEAR, /usr/local/lib64/python3.7/site-packages/paddle/vision/transforms/functional_pil.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead. 'bicubic': Image.BICUBIC, /usr/local/lib64/python3.7/site-packages/paddle/vision/transforms/functional_pil.py:39: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead. 'box': Image.BOX, /usr/local/lib64/python3.7/site-packages/paddle/vision/transforms/functional_pil.py:40: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead. 'lanczos': Image.LANCZOS, /usr/local/lib64/python3.7/site-packages/paddle/vision/transforms/functional_pil.py:41: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead. 'hamming': Image.HAMMING ----------- Configuration Arguments ----------- heter_worker_num: None heter_workers: http_port: None ips: 127.0.0.1 log_dir: ./ppyolo_dygraph/ nproc_per_node: None run_mode: None server_num: None servers: training_script: tools/train.py training_script_args: ['-c', 'configs/ppyolo/ppyolo_r50vd_dcn_1x_coco.yml'] worker_num: None workers: xpus: 0,1

WARNING 2022-04-22 16:04:55,019 launch.py:359] Not found distinct arguments and compiled with cuda or xpu. Default use collective mode launch train in XPU mode INFO 2022-04-22 16:04:55,037 launch_utils.py:510] Local start 2 processes. First process distributed environment info (Only For Debug): +=======================================================================================+ | Distributed Envs Value | +---------------------------------------------------------------------------------------+ | PADDLE_TRAINER_ID 0 | | PADDLE_CURRENT_ENDPOINT 127.0.0.1:51941 | | PADDLE_TRAINERS_NUM 2 | | PADDLE_TRAINER_ENDPOINTS 127.0.0.1:51941,127.0.0.1:38935 | | PADDLE_RANK_IN_NODE 0 | | PADDLE_LOCAL_DEVICE_IDS 0 | | PADDLE_WORLD_DEVICE_IDS 0,1 | | FLAGS_selected_accelerators 0 | | FLAGS_selected_xpus 0 | +=======================================================================================+

INFO 2022-04-22 16:04:55,038 launch_utils.py:514] details abouts PADDLE_TRAINER_ENDPOINTS can be found in ./ppyolo_dygraph//endpoints.log, and detail running logs maybe found in ./ppyolo_dygraph//workerlog.0 launch proc_id:4129 idx:0 launch proc_id:4132 idx:1 XPURT /usr/local/lib64/python3.7/site-packages/paddle/fluid/../libs/libxpurt.so loaded /usr/lib/python3.7/site-packages/setuptools/depends.py:2: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp /usr/local/lib64/python3.7/site-packages/paddle/vision/transforms/functional_pil.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead. 'nearest': Image.NEAREST, /usr/local/lib64/python3.7/site-packages/paddle/vision/transforms/functional_pil.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead. 'bilinear': Image.BILINEAR, /usr/local/lib64/python3.7/site-packages/paddle/vision/transforms/functional_pil.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead. 'bicubic': Image.BICUBIC, /usr/local/lib64/python3.7/site-packages/paddle/vision/transforms/functional_pil.py:39: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead. 'box': Image.BOX, /usr/local/lib64/python3.7/site-packages/paddle/vision/transforms/functional_pil.py:40: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead. 'lanczos': Image.LANCZOS, /usr/local/lib64/python3.7/site-packages/paddle/vision/transforms/functional_pil.py:41: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead. 'hamming': Image.HAMMING /usr/local/lib64/python3.7/site-packages/paddle/tensor/creation.py:125: DeprecationWarning: np.object is a deprecated alias for the builtin object. To silence this warning, use object by itself. Doing this will not modify any behavior and is safe. Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations if data.dtype == np.object: Traceback (most recent call last): File "tools/train.py", line 177, in main() File "tools/train.py", line 173, in main run(FLAGS, cfg) File "tools/train.py", line 112, in run init_parallel_env() File "/workspace/PaddleDetection/ppdet/engine/env.py", line 44, in init_parallel_env paddle.distributed.init_parallel_env() File "/usr/local/lib64/python3.7/site-packages/paddle/distributed/parallel.py", line 195, in init_parallel_env core.BKCLParallelContext(strategy, place)) AttributeError: module 'paddle.fluid.core_noavx' has no attribute 'BKCLParallelContext'

nemonameless commented 2 years ago

paddle安装成功与否,输出下卡数看看?另外配置文件除了数据路径外还有什么改动吗?

1chang37 commented 2 years ago

paddle 成功安装了,k200本身是一张卡,但是有2个xpu。配置文件只是修改了use_xpu