search

intel / torch-xpu-ops

Apache License 2.0
28 stars 19 forks source link

XPU backend: masked_select falling back to CPU #629

Closed uniartisan closed 2 months ago

uniartisan commented 2 months ago

🐛 Describe the bug

import torch

assert torch.xpu.is_available(), "Intel XPU is not available"

batch_size = 4
vocab_size = 4

out = torch.randn(batch_size, vocab_size, dtype=torch.bfloat16, device='xpu')
temperature = torch.full((batch_size,), 1.0, dtype=torch.bfloat16,device='xpu')  
top_p = torch.full((batch_size,), 0.8, dtype=torch.bfloat16, device='xpu')  

top_p_mask = (top_p > 0) & (top_p < 1)

try:
    log_probs = torch.nn.functional.log_softmax(
        out[top_p_mask] / temperature[top_p_mask].unsqueeze(1), dim=-1
    )
    print("Operation completed successfully")
except Exception as e:
    print(f"Operation failed with error: {e}")

print(f"top_p_mask shape: {top_p_mask.shape}")
print(f"Filtered out shape: {out[top_p_mask].shape}")
# print(f"log_probs shape: {log_probs.shape}")

masked_select operation falling back to CPU

This can not run correctly: log_probs = torch.nn.functional.log_softmax( out[top_p_mask] / temperature[top_p_mask].unsqueeze(1), dim=-1 )

UserWarning: The operator 'aten::masked_select on the XPU backend is falling back to run on the CPU. (Triggered internally at /home/lzy/workspace/pytorch/build/aten/src/ATen/xpu/RegisterXPU.cpp:6313.)

This warning indicates that the masked_select operation is not fully supported on the XPU backend and is falling back to CPU execution, which could potentially impact performance.

Proposed solution: Implement native XPU support for the masked_select operation to avoid CPU fallback.

Versions

Collecting environment information... PyTorch version: 2.5.0a0+git0022dc0 Is debug build: False CUDA used to build PyTorch: None ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64) GCC version: (conda-forge gcc 14.1.0-0) 14.1.0 Clang version: 14.0.0-1ubuntu1.1 CMake version: version 3.30.0 Libc version: glibc-2.35

Python version: 3.11.9 (main, Apr 19 2024, 16:48:06) [GCC 11.2.0] (64-bit runtime) Python platform: Linux-5.15.153.1-microsoft-standard-WSL2-x86_64-with-glibc2.35 Is CUDA available: False CUDA runtime version: No CUDA CUDA_MODULE_LOADING set to: N/A GPU models and configuration: No CUDA Nvidia driver version: No CUDA cuDNN version: No CUDA HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True

CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 39 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Vendor ID: GenuineIntel Model name: 13th Gen Intel(R) Core(TM) i7-13700KF CPU family: 6 Model: 183 Thread(s) per core: 2 Core(s) per socket: 8 Socket(s): 1 Stepping: 1 BogoMIPS: 6835.19 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid pni pclmulqdq vmx ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves avx_vnni umip waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize flush_l1d arch_capabilities Virtualization: VT-x Hypervisor vendor: Microsoft Virtualization type: full L1d cache: 384 KiB (8 instances) L1i cache: 256 KiB (8 instances) L2 cache: 16 MiB (8 instances) L3 cache: 30 MiB (1 instance) Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Retbleed: Mitigation; Enhanced IBRS Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected

Versions of relevant libraries: [pip3] flake8==7.1.0 [pip3] numpy==1.26.4 [pip3] optree==0.12.1 [pip3] torch==2.5.0a0+git0022dc0 [pip3] torchao==0.3.1 [pip3] triton==3.0.0 [conda] numpy 1.26.4 pypi_0 pypi [conda] optree 0.12.1 pypi_0 pypi [conda] torch 2.5.0a0+git0022dc0 dev_0 [conda] torchao 0.3.1 pypi_0 pypi [conda] triton 3.0.0 pypi_0 pypi

xytintel commented 2 months ago

PR ready: https://github.com/intel/torch-xpu-ops/pull/649