search

coreweave / ml-containers

MIT License
19 stars 3 forks source link

Triton may have been broken for a while in the nightly builds #64

Closed ad8e closed 4 months ago

ad8e commented 4 months ago

I receive this error:

[rank7]:     from triton.compiler.compiler import triton_key
[rank7]: torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
[rank7]: ImportError: cannot import name 'triton_key' from 'triton.compiler.compiler' (/usr/local/lib/python3.10/dist-packages/triton/compiler/compiler.py)

Same problem as this one: https://github.com/pytorch/pytorch/issues/123042

It's been present since before Mar 14. Given the solution in that thread, I suspect PyTorch's interaction with Triton may have changed. The non-nightly torch-extras build is fine.

Unfortunately, I have no testcase; running a simple torch.compile works fine. Maybe it'll solve itself when Pytorch 2.3 comes out; I'll bump this issue if it's still present when 2.3 lands in torch-extras.

ad8e commented 4 months ago

Update: our image, built on the stable torch-extras image, broke a few days ago. It is because pip's triton was bumped to 2.3.0. Pinning our own image's triton to 2.2.0 fixed the issue. So there is annoying interplay between pytorch and triton versions.

ad8e commented 4 months ago

Fixed, thanks Eta!