wangkl2
commented
1 month ago @ch0801 E external/intel_xla/xla/stream_executor/sycl/sycl_gpu_runtime.cc:178] Can not found any devices. From the output, the XPU devices are not detected via the framework.
Could you please run the following commands to collect the env info in your conda env, and provide us with the output? Thanks.
wget https://raw.githubusercontent.com/intel/intel-extension-for-tensorflow/main/tools/python/env_check.py
pip install wget
python env_check.py
ch0801
Failed to run intel-extension-for-tensorflow/tree/main/examples/train_maskrcnn on PVC (GPU Max 1550). Errors below showing "Can not found any devices." and " Failed precondition: No visible XPU devices". However I could run intel-extension-for-tensorflow/tree/main/examples/infer_resnet50 successfully on PVC. Also verified with "sycl-ls" that gpu 0 -7 existed.
2024-07-15 14:24:41.385979: I itex/core/wrapper/itex_gpu_wrapper.cc:38] Intel Extension for Tensorflow* GPU backend is loaded. 2024-07-15 14:24:41.387096: I external/local_xla/xla/pjrt/pjrt_api.cc:67] PJRT_Api is set for device type xpu 2024-07-15 14:24:41.387116: I external/local_xla/xla/pjrt/pjrt_api.cc:72] PJRT plugin for XPU has PJRT API version 0.33. The framework PJRT API version is 0.34. 2024-07-15 14:24:41.404971: E external/intel_xla/xla/stream_executor/sycl/sycl_gpu_runtime.cc:178] Can not found any devices. 2024-07-15 14:24:41.405019: E itex/core/kernels/xpu_kernel.cc:60] Failed precondition: No visible XPU devices. To check runtime environment on your host, please run itex/tools/python/env_check.py. If you need help, create an issue at https://github.com/intel/intel-extension-for-tensorflow/issues 2024-07-15 14:24:41.450735: E itex/core/devices/gpu/itex_gpu_runtime.cc:174] Can not found any devices. To check runtime environment on your host, please run itex/tools/python/env_check.py. If you need help, create an issue at https://github.com/intel/intel-extension-for-tensorflow/issues 2024-07-15 14:24:41,895 I dllogger PARAMETER image_size: (832, 1344), augment_input_data: True, gt_mask_size: 112, num_classes: 91, skip_crowd_during_training: True, use_category: True, rpn_positive_overlap: 0.7, rpn_negative_overlap: 0.3, rpn_batch_size_per_im: 256, rpn_fg_fraction: 0.5, rpn_min_size: 0.0, batch_size_per_im: 512, fg_fraction: 0.25, fg_thresh: 0.5, bg_thresh_hi: 0.5, bg_thresh_lo: 0.0, fast_rcnn_mlp_head_dim: 1024, bbox_reg_weights: (10.0, 10.0, 5.0, 5.0), include_mask: True, mrcnn_resolution: 28, train_rpn_pre_nms_topn: 2000, train_rpn_post_nms_topn: 1000, train_rpn_nms_threshold: 0.7, test_detections_per_image: 100, test_nms: 0.5, test_rpn_pre_nms_topn: 1000, test_rpn_post_nms_topn: 1000, test_rpn_nms_thresh: 0.7, min_level: 2, max_level: 6, num_scales: 1, aspect_ratios: [(1.0, 1.0), (1.4, 0.7), (0.7, 1.4)], anchor_scale: 8.0, rpn_box_loss_weight: 1.0, fast_rcnn_box_loss_weight: 1.0, mrcnn_weight_loss_mask: 1.0, checkpoint_name_format: nvidia_mrcnn_tf2.ckpt, mode: train, mpi_num: 0, data_dir: /nfs/site/home/chinglan/github/int-ext-tf-public/examples/train_maskrcnn/DeepLearningExamples/TensorFlow2/Segmentation/MaskRCNN/dataset/data, model_dir: /nfs/site/home/chinglan/github/int-ext-tf-public/examples/train_maskrcnn/DeepLearningExamples/TensorFlow2/Segmentation/MaskRCNN/output, backbone_checkpoint: None, eval_file: /data/annotations/instances_val2017.json, epochs: 12, steps_per_epoch: None, eval_samples: None, train_batch_size: 4, eval_batch_size: 8, seed: None, l2_weight_decay: 0.0001, init_learning_rate: 0.0, learning_rate_values: [0.01, 0.001, 0.0001], learning_rate_boundaries: [0.3, 8.0, 10.0], momentum: 0.9, finetune_bn: False, use_synthetic_data: False, xla: False, amp: False, log_file: mrcnn-dlll.json, log_every: 100, log_warmup_steps: 100, log_graph: False, log_tensorboard: None, verbose: False, eagerly: False 2024:07:15-14:24:42:(40388) |CCL_WARN| did not find MPI-launcher specific variables, switch to ATL/OFI, to force enable ATL/MPI set CCL_ATL_TRANSPORT=mpi 2024:07:15-14:24:42:(40388) |CCL_WARN| CCL_CONFIGURATION_PATH= is unknown to and unused by oneCCL code but is present in the environment, check if it is not mistyped. 2024:07:15-14:24:42:(40388) |CCL_WARN| could not get local_idx/count from environment variables, trying to get them from ATL Traceback (most recent call last): File "/nfs/site/home/chinglan/github/int-ext-tf-public/examples/train_maskrcnn/DeepLearningExamples/TensorFlow2/Segmentation/MaskRCNN/main.py", line 83, in
main()
File "/nfs/site/home/chinglan/github/int-ext-tf-public/examples/train_maskrcnn/DeepLearningExamples/TensorFlow2/Segmentation/MaskRCNN/main.py", line 68, in main
tf.config.experimental.set_visible_devices(xpus[hvd.local_rank()], 'XPU')
IndexError: list index out of range