AllenaiLongformerBase HF Train Float32 fails under LTS Driver

intel / intel-xpu-backend-for-triton

OpenAI Triton backend for Intel® GPUs

MIT License

109 stars 29 forks source link

AllenaiLongformerBase HF Train Float32 fails under LTS Driver #1336

Closed alexbaden closed 1 week ago

alexbaden commented 1 month ago

loading model: 0it [00:03, ?it/s]
xpu  train AllenaiLongformerBase      
  File "/localdisk/abaden/Projects/intel-xpu-backend-for-triton/python/triton/compiler/compiler.py", line 376, in _init_handles
    self.module, self.function, self.n_regs, self.n_spills = driver.active.utils.load_binary(
torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
RuntimeError: Triton Error [ZE]: 0x78000018

whitneywhtsang commented 1 month ago

https://github.com/intel/intel-xpu-backend-for-triton/pull/1337 disabled warpReduce on LTS to workaround the failure. This issue is to remove the workaround and properly fix it.