单机多卡训练报错 module 'paddle.fluid.libpaddle' has no attribute 'ProcessGroupNCCL'

1wang11lijian1 commented 1 year ago

请提供下述完整信息以便快速定位问题/Please provide the following information to quickly locate the problem

系统环境/System Environment：windows10, python3.7
版本号/Version：2.6 Paddle： PaddleOCR：问题相关组件/Related components：单机多卡训练报错 module 'paddle.fluid.libpaddle' has no attribute 'ProcessGroupNCCL'
运行指令/Command Code：python -m paddle.distributed.launch --gpus 0,1 tools/train.py -c configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_cml.yml -o Global.pretrain_weights=./pretrain_models/MobileNetV3_large_x0_5_pretrained/
完整报错/Complete Error Message： python -m paddle.distributed.launch --gpus 0,1 tools/train.py -c configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_cml.yml -o Global.pretrain_weights=./pretrain_models/MobileNetV3_large_x0_5_pretrained/ LAUNCH INFO 2023-03-06 12:37:00,361 ----------- Configuration ---------------------- LAUNCH INFO 2023-03-06 12:37:00,361 devices: 0,1 LAUNCH INFO 2023-03-06 12:37:00,362 elastic_level: -1 LAUNCH INFO 2023-03-06 12:37:00,362 elastic_timeout: 30 LAUNCH INFO 2023-03-06 12:37:00,362 gloo_port: 6767 LAUNCH INFO 2023-03-06 12:37:00,362 host: None LAUNCH INFO 2023-03-06 12:37:00,362 ips: None LAUNCH INFO 2023-03-06 12:37:00,362 job_id: default LAUNCH INFO 2023-03-06 12:37:00,362 legacy: False LAUNCH INFO 2023-03-06 12:37:00,362 log_dir: log LAUNCH INFO 2023-03-06 12:37:00,362 log_level: INFO LAUNCH INFO 2023-03-06 12:37:00,363 master: None LAUNCH INFO 2023-03-06 12:37:00,363 max_restart: 3 LAUNCH INFO 2023-03-06 12:37:00,363 nnodes: 1 LAUNCH INFO 2023-03-06 12:37:00,363 nproc_per_node: None LAUNCH INFO 2023-03-06 12:37:00,363 rank: -1 LAUNCH INFO 2023-03-06 12:37:00,363 run_mode: collective LAUNCH INFO 2023-03-06 12:37:00,363 server_num: None LAUNCH INFO 2023-03-06 12:37:00,363 servers: LAUNCH INFO 2023-03-06 12:37:00,363 start_port: 6070 LAUNCH INFO 2023-03-06 12:37:00,363 trainer_num: None LAUNCH INFO 2023-03-06 12:37:00,364 trainers: LAUNCH INFO 2023-03-06 12:37:00,364 training_script: tools/train.py LAUNCH INFO 2023-03-06 12:37:00,364 training_script_args: ['-c', 'configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_cml.yml', '-o', 'Global.pretrain_weights=./pretrain_models/MobileNetV3_large_x0_5_pretrained/'] LAUNCH INFO 2023-03-06 12:37:00,364 with_gloo: 1 LAUNCH INFO 2023-03-06 12:37:00,364 -------------------------------------------------- LAUNCH INFO 2023-03-06 12:37:00,365 Job: default, mode collective, replicas 1[1:1], elastic False LAUNCH INFO 2023-03-06 12:37:00,367 Run Pod: zupssk, replicas 2, status ready LAUNCH INFO 2023-03-06 12:37:00,374 Watching Pod: zupssk, replicas 2, status running LAUNCH WARNING 2023-03-06 12:37:00,500 save gpu info failed [2023/03/06 12:37:02] ppocr INFO: Architecture : [2023/03/06 12:37:02] ppocr INFO: Models : [2023/03/06 12:37:02] ppocr INFO: Student : [2023/03/06 12:37:02] ppocr INFO: Backbone : [2023/03/06 12:37:02] ppocr INFO: disable_se : True [2023/03/06 12:37:02] ppocr INFO: model_name : large [2023/03/06 12:37:02] ppocr INFO: name : MobileNetV3 [2023/03/06 12:37:02] ppocr INFO: scale : 0.5 [2023/03/06 12:37:02] ppocr INFO: Head : [2023/03/06 12:37:02] ppocr INFO: k : 50 [2023/03/06 12:37:02] ppocr INFO: name : DBHead [2023/03/06 12:37:02] ppocr INFO: Neck : [2023/03/06 12:37:02] ppocr INFO: name : RSEFPN [2023/03/06 12:37:02] ppocr INFO: out_channels : 96 [2023/03/06 12:37:02] ppocr INFO: shortcut : True [2023/03/06 12:37:02] ppocr INFO: Transform : None [2023/03/06 12:37:02] ppocr INFO: algorithm : DB [2023/03/06 12:37:02] ppocr INFO: model_type : det [2023/03/06 12:37:02] ppocr INFO: pretrained : None [2023/03/06 12:37:02] ppocr INFO: Student2 : [2023/03/06 12:37:02] ppocr INFO: Backbone : [2023/03/06 12:37:02] ppocr INFO: disable_se : True [2023/03/06 12:37:02] ppocr INFO: model_name : large [2023/03/06 12:37:02] ppocr INFO: name : MobileNetV3 [2023/03/06 12:37:02] ppocr INFO: scale : 0.5 [2023/03/06 12:37:02] ppocr INFO: Head : [2023/03/06 12:37:02] ppocr INFO: k : 50 [2023/03/06 12:37:02] ppocr INFO: name : DBHead [2023/03/06 12:37:02] ppocr INFO: Neck : [2023/03/06 12:37:02] ppocr INFO: name : RSEFPN [2023/03/06 12:37:02] ppocr INFO: out_channels : 96 [2023/03/06 12:37:02] ppocr INFO: shortcut : True [2023/03/06 12:37:02] ppocr INFO: Transform : None [2023/03/06 12:37:02] ppocr INFO: algorithm : DB [2023/03/06 12:37:02] ppocr INFO: model_type : det [2023/03/06 12:37:02] ppocr INFO: pretrained : None [2023/03/06 12:37:02] ppocr INFO: Teacher : [2023/03/06 12:37:02] ppocr INFO: Backbone : [2023/03/06 12:37:02] ppocr INFO: in_channels : 3 [2023/03/06 12:37:02] ppocr INFO: layers : 50 [2023/03/06 12:37:02] ppocr INFO: name : ResNet_vd [2023/03/06 12:37:02] ppocr INFO: Head : [2023/03/06 12:37:02] ppocr INFO: k : 50 [2023/03/06 12:37:02] ppocr INFO: kernel_list : [7, 2, 2] [2023/03/06 12:37:02] ppocr INFO: name : DBHead [2023/03/06 12:37:02] ppocr INFO: Neck : [2023/03/06 12:37:02] ppocr INFO: name : LKPAN [2023/03/06 12:37:02] ppocr INFO: out_channels : 256 [2023/03/06 12:37:02] ppocr INFO: algorithm : DB [2023/03/06 12:37:02] ppocr INFO: freeze_params : True [2023/03/06 12:37:02] ppocr INFO: model_type : det [2023/03/06 12:37:02] ppocr INFO: pretrained : None [2023/03/06 12:37:02] ppocr INFO: return_all_feats : False [2023/03/06 12:37:02] ppocr INFO: algorithm : Distillation [2023/03/06 12:37:02] ppocr INFO: model_type : det [2023/03/06 12:37:02] ppocr INFO: name : DistillationModel [2023/03/06 12:37:02] ppocr INFO: Eval : [2023/03/06 12:37:02] ppocr INFO: dataset : [2023/03/06 12:37:02] ppocr INFO: data_dir : ./train_data/CY_data_5000/det/ [2023/03/06 12:37:02] ppocr INFO: label_file_list : ['./train_data/CY_data_5000/det/test.txt'] [2023/03/06 12:37:02] ppocr INFO: name : SimpleDataSet [2023/03/06 12:37:02] ppocr INFO: transforms : [2023/03/06 12:37:02] ppocr INFO: DecodeImage : [2023/03/06 12:37:02] ppocr INFO: channel_first : False [2023/03/06 12:37:02] ppocr INFO: img_mode : BGR [2023/03/06 12:37:02] ppocr INFO: DetLabelEncode : None [2023/03/06 12:37:02] ppocr INFO: DetResizeForTest : None [2023/03/06 12:37:02] ppocr INFO: NormalizeImage : [2023/03/06 12:37:02] ppocr INFO: mean : [0.485, 0.456, 0.406] [2023/03/06 12:37:02] ppocr INFO: order : hwc [2023/03/06 12:37:02] ppocr INFO: scale : 1./255. [2023/03/06 12:37:02] ppocr INFO: std : [0.229, 0.224, 0.225] [2023/03/06 12:37:02] ppocr INFO: ToCHWImage : None [2023/03/06 12:37:02] ppocr INFO: KeepKeys : [2023/03/06 12:37:02] ppocr INFO: keep_keys : ['image', 'shape', 'polys', 'ignore_tags'] [2023/03/06 12:37:02] ppocr INFO: loader : [2023/03/06 12:37:02] ppocr INFO: batch_size_per_card : 1 [2023/03/06 12:37:02] ppocr INFO: drop_last : False [2023/03/06 12:37:02] ppocr INFO: num_workers : 2 [2023/03/06 12:37:02] ppocr INFO: shuffle : False [2023/03/06 12:37:02] ppocr INFO: Global : [2023/03/06 12:37:02] ppocr INFO: cal_metric_during_train : False [2023/03/06 12:37:02] ppocr INFO: checkpoints : None [2023/03/06 12:37:02] ppocr INFO: debug : False [2023/03/06 12:37:02] ppocr INFO: distributed : True [2023/03/06 12:37:02] ppocr INFO: epoch_num : 500 [2023/03/06 12:37:02] ppocr INFO: eval_batch_step : [0, 400] [2023/03/06 12:37:02] ppocr INFO: infer_img : None [2023/03/06 12:37:02] ppocr INFO: log_smooth_window : 20 [2023/03/06 12:37:02] ppocr INFO: pretrain_weights : ./pretrain_models/MobileNetV3_large_x0_5_pretrained/ [2023/03/06 12:37:02] ppocr INFO: pretrained_model : None [2023/03/06 12:37:02] ppocr INFO: print_batch_step : 10 [2023/03/06 12:37:02] ppocr INFO: save_epoch_step : 100 [2023/03/06 12:37:02] ppocr INFO: save_inference_dir : None [2023/03/06 12:37:02] ppocr INFO: save_model_dir : ./output/en_PP-OCR_v3_det/ [2023/03/06 12:37:02] ppocr INFO: save_res_path : ./output/det/predicts_ppocrv3_en.txt [2023/03/06 12:37:02] ppocr INFO: use_gpu : True [2023/03/06 12:37:02] ppocr INFO: use_visualdl : False [2023/03/06 12:37:02] ppocr INFO: Loss : [2023/03/06 12:37:02] ppocr INFO: loss_config_list : [2023/03/06 12:37:02] ppocr INFO: DistillationDilaDBLoss : [2023/03/06 12:37:02] ppocr INFO: alpha : 5 [2023/03/06 12:37:02] ppocr INFO: balance_loss : True [2023/03/06 12:37:02] ppocr INFO: beta : 10 [2023/03/06 12:37:02] ppocr INFO: key : maps [2023/03/06 12:37:02] ppocr INFO: main_loss_type : DiceLoss [2023/03/06 12:37:02] ppocr INFO: model_name_pairs : [['Student', 'Teacher'], ['Student2', 'Teacher']] [2023/03/06 12:37:02] ppocr INFO: ohem_ratio : 3 [2023/03/06 12:37:02] ppocr INFO: weight : 1.0 [2023/03/06 12:37:02] ppocr INFO: DistillationDMLLoss : [2023/03/06 12:37:02] ppocr INFO: key : maps [2023/03/06 12:37:02] ppocr INFO: maps_name : thrink_maps [2023/03/06 12:37:02] ppocr INFO: model_name_pairs : ['Student', 'Student2'] [2023/03/06 12:37:02] ppocr INFO: weight : 1.0 [2023/03/06 12:37:02] ppocr INFO: DistillationDBLoss : [2023/03/06 12:37:02] ppocr INFO: alpha : 5 [2023/03/06 12:37:02] ppocr INFO: balance_loss : True [2023/03/06 12:37:02] ppocr INFO: beta : 10 [2023/03/06 12:37:02] ppocr INFO: main_loss_type : DiceLoss [2023/03/06 12:37:02] ppocr INFO: model_name_list : ['Student', 'Student2'] [2023/03/06 12:37:02] ppocr INFO: ohem_ratio : 3 [2023/03/06 12:37:02] ppocr INFO: weight : 1.0 [2023/03/06 12:37:02] ppocr INFO: name : CombinedLoss [2023/03/06 12:37:02] ppocr INFO: Metric : [2023/03/06 12:37:02] ppocr INFO: base_metric_name : DetMetric [2023/03/06 12:37:02] ppocr INFO: key : Student [2023/03/06 12:37:02] ppocr INFO: main_indicator : hmean [2023/03/06 12:37:02] ppocr INFO: name : DistillationMetric [2023/03/06 12:37:02] ppocr INFO: Optimizer : [2023/03/06 12:37:02] ppocr INFO: beta1 : 0.9 [2023/03/06 12:37:02] ppocr INFO: beta2 : 0.999 [2023/03/06 12:37:02] ppocr INFO: lr : [2023/03/06 12:37:02] ppocr INFO: learning_rate : 0.001 [2023/03/06 12:37:02] ppocr INFO: name : Cosine [2023/03/06 12:37:02] ppocr INFO: warmup_epoch : 2 [2023/03/06 12:37:02] ppocr INFO: name : Adam [2023/03/06 12:37:02] ppocr INFO: regularizer : [2023/03/06 12:37:02] ppocr INFO: factor : 5e-05 [2023/03/06 12:37:02] ppocr INFO: name : L2 [2023/03/06 12:37:02] ppocr INFO: PostProcess : [2023/03/06 12:37:02] ppocr INFO: box_thresh : 0.6 [2023/03/06 12:37:02] ppocr INFO: key : head_out [2023/03/06 12:37:02] ppocr INFO: max_candidates : 1000 [2023/03/06 12:37:02] ppocr INFO: model_name : ['Student'] [2023/03/06 12:37:02] ppocr INFO: name : DistillationDBPostProcess [2023/03/06 12:37:02] ppocr INFO: thresh : 0.3 [2023/03/06 12:37:02] ppocr INFO: unclip_ratio : 1.5 [2023/03/06 12:37:02] ppocr INFO: Train : [2023/03/06 12:37:02] ppocr INFO: dataset : [2023/03/06 12:37:02] ppocr INFO: data_dir : ./train_data/CY_data_5000/det/ [2023/03/06 12:37:02] ppocr INFO: label_file_list : ['./train_data/CY_data_5000/det/train.txt'] [2023/03/06 12:37:02] ppocr INFO: name : SimpleDataSet [2023/03/06 12:37:02] ppocr INFO: ratio_list : [1.0] [2023/03/06 12:37:02] ppocr INFO: transforms : [2023/03/06 12:37:02] ppocr INFO: DecodeImage : [2023/03/06 12:37:02] ppocr INFO: channel_first : False [2023/03/06 12:37:02] ppocr INFO: img_mode : BGR [2023/03/06 12:37:02] ppocr INFO: DetLabelEncode : None [2023/03/06 12:37:02] ppocr INFO: CopyPaste : None [2023/03/06 12:37:02] ppocr INFO: IaaAugment : [2023/03/06 12:37:02] ppocr INFO: augmenter_args : [2023/03/06 12:37:02] ppocr INFO: args : [2023/03/06 12:37:02] ppocr INFO: p : 0.5 [2023/03/06 12:37:02] ppocr INFO: type : Fliplr [2023/03/06 12:37:02] ppocr INFO: args : [2023/03/06 12:37:02] ppocr INFO: rotate : [-10, 10] [2023/03/06 12:37:02] ppocr INFO: type : Affine [2023/03/06 12:37:02] ppocr INFO: args : [2023/03/06 12:37:02] ppocr INFO: size : [0.5, 3] [2023/03/06 12:37:02] ppocr INFO: type : Resize [2023/03/06 12:37:02] ppocr INFO: EastRandomCropData : [2023/03/06 12:37:02] ppocr INFO: keep_ratio : True [2023/03/06 12:37:02] ppocr INFO: max_tries : 50 [2023/03/06 12:37:02] ppocr INFO: size : [960, 960] [2023/03/06 12:37:02] ppocr INFO: MakeBorderMap : [2023/03/06 12:37:02] ppocr INFO: shrink_ratio : 0.4 [2023/03/06 12:37:02] ppocr INFO: thresh_max : 0.7 [2023/03/06 12:37:02] ppocr INFO: thresh_min : 0.3 [2023/03/06 12:37:02] ppocr INFO: MakeShrinkMap : [2023/03/06 12:37:02] ppocr INFO: min_text_size : 8 [2023/03/06 12:37:02] ppocr INFO: shrink_ratio : 0.4 [2023/03/06 12:37:02] ppocr INFO: NormalizeImage : [2023/03/06 12:37:02] ppocr INFO: mean : [0.485, 0.456, 0.406] [2023/03/06 12:37:02] ppocr INFO: order : hwc [2023/03/06 12:37:02] ppocr INFO: scale : 1./255. [2023/03/06 12:37:02] ppocr INFO: std : [0.229, 0.224, 0.225] [2023/03/06 12:37:02] ppocr INFO: ToCHWImage : None [2023/03/06 12:37:02] ppocr INFO: KeepKeys : [2023/03/06 12:37:02] ppocr INFO: keep_keys : ['image', 'threshold_map', 'threshold_mask', 'shrink_map', 'shrink_mask'] [2023/03/06 12:37:02] ppocr INFO: loader : [2023/03/06 12:37:02] ppocr INFO: batch_size_per_card : 6 [2023/03/06 12:37:02] ppocr INFO: drop_last : False [2023/03/06 12:37:02] ppocr INFO: num_workers : 2 [2023/03/06 12:37:02] ppocr INFO: shuffle : True [2023/03/06 12:37:02] ppocr INFO: profiler_options : None [2023/03/06 12:37:02] ppocr INFO: train with paddle 2.4.2 and device Place(gpu:0) I0306 12:37:02.999004 22968 tcp_utils.cc:181] The server starts to listen on IP_ANY:54344 I0306 12:37:02.999004 22968 tcp_utils.cc:130] Successfully connected to 192.168.110.129:54344 Traceback (most recent call last): File "tools/train.py", line 202, in main(config, device, logger, vdl_writer) File "tools/train.py", line 47, in main dist.init_parallel_env() File "C:\ProgramData\Anaconda3\envs\PPOCR_env\lib\site-packages\paddle\distributed\parallel.py", line 287, in init_parallel_env pg_options=None, File "C:\ProgramData\Anaconda3\envs\PPOCR_env\lib\site-packages\paddle\distributed\collective.py", line 207, in _new_process_group_impl pg = core.ProcessGroupNCCL(store, rank, world_size, place, group_id) AttributeError: module 'paddle.fluid.libpaddle' has no attribute 'ProcessGroupNCCL' LAUNCH INFO 2023-03-06 12:37:04,425 Pod failed LAUNCH ERROR 2023-03-06 12:37:04,425 Container failed !!! Container rank 1 status failed cmd ['C:\ProgramData\Anaconda3\envs\PPOCR_env\python.exe', '-u', 'tools/train.py', '-c', 'configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_cml.yml', '-o', 'Global.pretrain_weights=./pretrain_models/MobileN etV3_large_x0_5_pretrained/'] code 1 log log\workerlog.1 env {'TERM_SESSION_ID': '1c16e761-04e2-4659-819c-66d9780616f3', 'COMMONPROGRAMW6432': 'C:\Program Files\Common Files', 'PROGRAMW6432': 'C:\Program Files', 'CONDA_DEFAULT_ENV': 'PPOCR_env', 'CONDA_SHLVL': '1', 'USERNAME': 'user', 'ALLUSERSPROFILE': 'C:\ProgramData', 'USERPROFILE': 'C:\Users\user', 'CUDA_PATH_V11_1': 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1', 'PROCESSOR_REVISION': '3100', 'IDEA_INITIAL_DIRECTORY': 'C:\Users\user\Des ktop', 'FPS_BROWSER_APP_PROFILE_STRING': 'Internet Explorer', 'PUBLIC': 'C:\Users\Public', 'PATH': 'C:\ProgramData\Anaconda3\envs\PPOCR_env\lib\site-packages\cv2\../../x64/vc14/bin;C:\ProgramData\Anaconda3\envs\PPOCR_e nv\lib\site-packages\paddle\fluid;C:\ProgramData\Anaconda3\envs\PPOCR_env\lib\site-packages\paddle\fluid\..\libs;C:\ProgramData\Anaconda3\envs\PPOCR_env;C:\ProgramData\Anaconda3\envs\PPOCR_env\Library\mingw-w 64\bin;C:\ProgramData\Anaconda3\envs\PPOCR_env\Library\usr\bin;C:\ProgramData\Anaconda3\envs\PPOCR_env\Library\bin;C:\ProgramData\Anaconda3\envs\PPOCR_env\Scripts;C:\ProgramData\Anaconda3\envs\PPOCR_env\bin;C :\ProgramData\Anaconda3\condabin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\libnvvp;C:\ProgramData\Anaconda3;C:\ProgramData\Anaconda3\Libra ry\mingw-w64\bin;C:\ProgramData\Anaconda3\Library\usr\bin;C:\ProgramData\Anaconda3\Library\bin;C:\ProgramData\Anaconda3\Scripts;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\Windo wsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\Program Files\MATLAB\R2018b\runtime\win64;C:\Program Files\MATLAB\R2018b\bin;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\WINDOWS\system32;C:\WINDOWS; C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0;C:\WINDOWS\System32\OpenSSH;\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\lib\ x64;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\libnvvp;C:\Program Files\dotnet;C:\Program Files\Microsoft SQL Server\130\Tools\Binn;C:\Prog ram Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn;D:\鏉烆垯娆\ue542Git\cmd;D:\鏉烆垯娆\ue542Git\mingw64\libexec\git-core;D:\鏉烆垯娆\ue542Git\bin;D:\鏉烆垯娆\ue542Git;C:\Program Files\Bandizip;C:\Users\ \user\AppData\Local\Microsoft\WindowsApps;.;D:\Modelsim\win64;C:\Program Files\JetBrains\PyCharm Community Edition 2021.2.3\bin;.;C:\Users\user\.dotnet\tools;C:\Users\user\AppData\Local\Programs\Microsoft VS Code \bin;C:\ProgramData\Anaconda3\envs\PPOCR_env\lib\site-packages\numpy\.libs', 'DRIVERDATA': 'C:\Windows\System32\Drivers\DriverData', 'HOMEDRIVE': 'C:', 'SESSIONNAME': 'Console', 'LOGONSERVER': '\\DESKTOP-GHMOIDO', 'TE RMINAL_EMULATOR': 'JetBrains-JediTerm', 'CONDA_PREFIX': 'C:\ProgramData\Anaconda3\envs\PPOCR_env', 'HOMEPATH': '\Users\user', 'SYSTEMROOT': 'C:\WINDOWS', 'SOPC_KIT_NIOS2': 'D:\quartus13\nios2eds', 'LOCALAPPDATA': 'C:\Users \user\AppData\Local', 'APPDATA': 'C:\Users\user\AppData\Roaming', 'PROCESSOR_IDENTIFIER': 'AMD64 Family 23 Model 49 Stepping 0, AuthenticAMD', 'PATHEXT': '.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC', 'PSMODULEPATH' : 'C:\Program Files\WindowsPowerShell\Modules;C:\WINDOWS\system32\WindowsPowerShell\v1.0\Modules', 'CONDA_PROMPT_MODIFIER': '(PPOCR_env) ', 'PROGRAMFILES(X86)': 'C:\Program Files (x86)', 'PROMPT': '(PPOCR_env) $P$G', 'NVTOO LSEXT_PATH': 'C:\Program Files\NVIDIA Corporation\NvToolsExt\', 'OS': 'Windows_NT', 'PROCESSOR_ARCHITECTURE': 'AMD64', 'NUMBER_OF_PROCESSORS': '64', 'COMSPEC': 'C:\WINDOWS\system32\cmd.exe', 'PROCESSOR_LEVEL': '23', 'PYCHARM COMMUNITY EDITION': 'C:\Program Files\JetBrains\PyCharm Community Edition 2021.2.3\bin;', 'USERDOMAIN_ROAMINGPROFILE': 'DESKTOP-GHMOIDO', 'MGLS_LICENSE_FILE': 'D:\Modelsim\LICENSE.TXT', 'PROGRAMFILES': 'C:\Program Files', 'WI NDIR': 'C:\WINDOWS', 'TEMP': 'C:\Users\user\AppData\Local\Temp', 'TMP': 'C:\Users\user\AppData\Local\Temp', 'COMMONPROGRAMFILES(X86)': 'C:\Program Files (x86)\Common Files', 'ONEDRIVE': 'C:\Users\user\OneDrive', 'CUD A_PATH': 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1', 'USERDOMAIN': 'DESKTOP-GHMOIDO', 'SYSTEMDRIVE': 'C:', 'COMPUTERNAME': 'DESKTOP-GHMOIDO', 'PROGRAMDATA': 'C:\ProgramData', 'NVCUDASAMPLES_ROOT': 'C:\ProgramDa ta\NVIDIA Corporation\CUDA Samples\v11.1', 'FPS_BROWSER_USER_PROFILE_STRING': 'Default', 'NVCUDASAMPLES11_1_ROOT': 'C:\ProgramData\NVIDIA Corporation\CUDA Samples\v11.1', 'COMMONPROGRAMFILES': 'C:\Program Files\Common Files ', 'QUARTUS_ROOTDIR': 'D:\quartus13\quartus', 'INTELLIJ_COMMAND_HISTFILE': 'C:\Users\user\AppData\Local\JetBrains\PyCharmCE2022.3\terminal\history\PaddleOCR-release-2.6-history6', 'CUSTOM_DEVICE_ROOT': '', 'OMP_NUM_TH READS': '1', 'POD_NAME': 'zupssk', 'PADDLE_MASTER': '192.168.110.129:54344', 'PADDLE_GLOBAL_SIZE': '2', 'PADDLE_LOCAL_SIZE': '2', 'PADDLE_GLOBAL_RANK': '1', 'PADDLE_LOCAL_RANK': '1', 'PADDLE_NNODES': '1', 'PADDLE_TRAINER_ENDPOINTS': '192.168.110.129:54345,192.168.110.129:54346', 'PADDLE_CURRENT_ENDPOINT': '192.168.110.129:54346', 'PADDLE_TRAINER_ID': '1', 'PADDLE_TRAINERS_NUM': '2', 'PADDLE_RANK_IN_NODE': '1', 'FLAGS_selected_gpus': '1'} LAUNCH INFO 2023-03-06 12:37:04,427 ------------------------- ERROR LOG DETAIL ------------------------- I0306 12:37:03.501117 25900 tcp_utils.cc:130] Successfully connected to 192.168.110.129:54344 Traceback (most recent call last): File "tools/train.py", line 202, in main(config, device, logger, vdl_writer) File "tools/train.py", line 47, in main dist.init_parallel_env() File "C:\ProgramData\Anaconda3\envs\PPOCR_env\lib\site-packages\paddle\distributed\parallel.py", line 287, in init_parallel_env pg_options=None, File "C:\ProgramData\Anaconda3\envs\PPOCR_env\lib\site-packages\paddle\distributed\collective.py", line 207, in _new_process_group_impl pg = core.ProcessGroupNCCL(store, rank, world_size, place, group_id) AttributeError: module 'paddle.fluid.libpaddle' has no attribute 'ProcessGroupNCCL' LAUNCH INFO 2023-03-06 12:37:04,637 Exit code 1

LDOUBLEV commented 1 year ago

参考这个API，验证paddle是否正确安装了https://www.paddlepaddle.org.cn/documentation/docs/zh/api/paddle/utils/run_check_cn.html#run-check

1wang11lijian1 commented 1 year ago

OK，谢谢你的回答，下面是我运行的结果，请问这个报错说用不了并行运算什么意思

python Python 3.7.16 (default, Jan 17 2023, 16:06:28) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32 Type "help", "copyright", "credits" or "license" for more information.

import paddle

paddle.utils.run_check() Running verify PaddlePaddle program ... W0307 10:05:31.748668 24496 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 8.6, Driver API Version: 11.2, Runtime API Version: 11.2 W0307 10:05:31.753669 24496 gpu_resources.cc:91] device: 0, cuDNN Version: 8.0. PaddlePaddle works well on 1 GPU. C:\ProgramData\Anaconda3\envs\PPOCR_env\lib\site-packages\paddle\fluid\executor.py:1585: UserWarning: Standalone executor is not used for data parallel UserWarning) W0307 10:05:35.871594 24496 parallel_executor.cc:666] Cannot enable P2P access from 0 to 1 W0307 10:05:35.871594 24496 parallel_executor.cc:666] Cannot enable P2P access from 1 to 0 WARNING:root:PaddlePaddle meets some problem with 2 GPUs. This may be caused by:

There is not enough GPUs visible on your system

Some GPUs are occupied by other process now

NVIDIA-NCCL2 is not installed correctly on your system. Please follow instruction on https://github.com/NVIDIA/nccl-tests to test your NCCL, or reinstall it following https://docs.nvidia.com/deeplearning/sdk/nccl-install-guide/index.html WARNING:root: Original Error is: (Unavailable) Windows can support Single GPU only. [Hint: Expected device_count == 1, but received device_count:2 != 1:1.] (at ..\paddle\fluid\framework\parallel_executor.cc:1322)

PaddlePaddle is installed successfully ONLY for single GPU! Let's start deep learning with PaddlePaddle now.

LDOUBLEV commented 1 year ago

可能是你的机器没有安装NCCL，或者paddle没有找到nccl的路径；按照提示3.检查下nccl是否正确安装了

happybear1015 commented 1 year ago

请问你解决了吗? 我也遇到了同样的问题。

1wang11lijian1 commented 1 year ago

@happybear1015 你好我查了很多资料发现多卡训练只能在linux平台下，windows平台只能进行单卡训练

OpenOneV commented 1 year ago

可能是你的机器没有安装NCCL，或者paddle没有找到nccl的路径；按照提示3.检查下nccl是否正确安装了

@happybear1015 你好我查了很多资料发现多卡训练只能在linux平台下，windows平台只能进行单卡训练

@LDOUBLEV

请问是真的吗？单卡多机也必须要NCCL框架吗？并且因为NNCL不支持的关系，Windows没法多机训练吗？（Linux的问题是驱动支持不好，显卡奇怪一点，Linux下的显卡驱动就不支持了）

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

PaddlePaddle / PaddleOCR

单机多卡训练报错 module 'paddle.fluid.libpaddle' has no attribute 'ProcessGroupNCCL' #9313