search

triton-lang / triton

Development repository for the Triton language and compiler
https://triton-lang.org/
MIT License
12.63k stars 1.53k forks source link

triton build failing to access https://tritonlang.blob.core.windows.net/llvm-builds/ with HTTP Error 409 #4527

Open lisaong opened 3 weeks ago

lisaong commented 3 weeks ago

This was working earlier, so recently happened.

PublicAccessNotPermitted Public access is not permitted on this storage account. RequestId:372d399d-901e-0036-4086-ef20cc000000 Time:2024-08-16T02:46:44.4328589Z 
[] #186 27.76 Building wheels for collected packages: triton
[] #186 27.76   Building wheel for triton (pyproject.toml): started
[] #186 28.67   error: subprocess-exited-with-error
[] #186 28.67   
[] #186 28.67   × Building wheel for triton (pyproject.toml) did not run successfully.
[] #186 28.67   │ exit code: 1
[] #186 28.67   ╰─> [7 lines of output]
[] #186 28.67       running bdist_wheel
[] #186 28.67       running build
[] #186 28.67       running build_py
[] #186 28.67       running build_ext
[] #186 28.67       downloading and extracting https://github.com/pybind/pybind11/archive/refs/tags/v2.11.1.tar.gz ...
[] #186 28.67       downloading and extracting https://tritonlang.blob.core.windows.net/llvm-builds/llvm-49af6502-ubuntu-x64.tar.gz ...
[] #186 28.67       error: HTTP Error 409: Public access is not permitted on this storage account.
[] #186 28.67       [end of output]
prarit commented 3 weeks ago

I'm seeing this too. The link to the tarball is now restricted.

prarit commented 3 weeks ago

In case anyone is looking into this: I've tried accessing https://tritonlang.blob.core.windows.net via an internal redhat.com network, a private network and via an online anonymizer. It looks like the site is locked down for some reason and access to the tarball is restricted as indicated in the initial report by @lisaong

prarit commented 3 weeks ago

Is there a way to provide a 1-time download for the tarball? Perhaps through a direct commit to the triton repo itself?

atalman commented 3 weeks ago

We are also seeing this when trying to build triton in pytorch: https://github.com/pytorch/pytorch/actions/runs/10421322991/job/28863284428

cc @ptillet @Jokeren

jayfurmanek commented 3 weeks ago

That URL was changed back in June: https://github.com/triton-lang/triton/commit/06e6799f4eba6035ec35c528e8fefd3d4d724b6f

Perhaps torch is on an older commit?

ytaous commented 3 weeks ago

The url is till blocked - any updates?

TheTrustedComputer commented 3 weeks ago

I also faced this HTTP 409 error when building PyTorch 2.4 + ROCm. I think this problem of Triton not building points to AOTriton (AMD's fork) using the old upstream commit containing the now broken URL.

fei-xx commented 3 weeks ago

I'm also seeing the similar issue. Any updates?

cantor-set commented 3 weeks ago

This is still an issue, AOTriton still can not be build

prarit commented 3 weeks ago

For anyone attempting to build aotriton from ROCm. aotriton references a DIFFERENT repository than triton-lang, and I have filed an issue here: https://github.com/ROCm/triton/issues/631

hyhuang00 commented 3 weeks ago

That URL was changed back in June: 06e6799

Perhaps torch is on an older commit?

Thank you! Using the new URL solved this problem.

prarit commented 3 weeks ago

@jayfurmanek torch is indeed on an older commit. Is there any plan to provide the older llvm-49af6502-ubuntu-x64.tar.gz tarball at the new oatriton download link? If not, is there a way I and others could get a copy?

vaenyr commented 3 weeks ago

For anyone who might be in a similar situation:

I've run into the same issue recently when trying to build PT for ROCm from source. I've tried a couple of ideas, including upgrading LLVM in AMD's triton fork, etc., all in vain. In the end what worked well for me was to follow the Building with custom LLVM part of README.

  1. Clone and checkout LLVM to commit 49af6502c6dcb4a7f7520178bd14df396f78240c.
  2. Build with AMDGPU support
  3. set the env variables as explained in the readme (I used export ...)
  4. build pytorch as usual

No more HTTP 409/missing blobs/version mismatch problems - everything went smoothly.

prarit commented 3 weeks ago

Thanks. I had actually stumbled across this today and was trying it out :) It seems like it works -- I'm just worried that the AMD llvm may have additional flags we're not compiling with. Hopefully someone from the AMD ROCm team can confirm the above instructions are valid.

TheTrustedComputer commented 2 weeks ago

I also had mixed results on building Triton (upstream commit cb3d79a185e40c9d8a579bea07747a8a8d157d52 and AMD's triton commit 9b73a543a5545960bcaf2830900b0560eec443c5) with this particular LLVM commit (49af6502c6dcb4a7f7520178bd14df396f78240c). I seem to cannot get past this error installing the Python package:

CMake Error at CMakeLists.txt:61 (find_package):
Found package configuration file:

    /root/triton-llvm-build/lib/cmake/mlir/MLIRConfig.cmake

but it set MLIR_FOUND to FALSE so package "MLIR" is considered to be NOT
FOUND.  Reason given by package:

The following imported targets are referenced, but are missing:
LLVMNVPTXCodeGen LLVMNVPTXDesc LLVMNVPTXInfo

-- Configuring incomplete, errors occurred

I built LLVM with CMake using the following flags:

-B /root/triton-llvm-build \
-DCMAKE_BUILD_TYPE=Release \
-DLLVM_DEFAULT_TARGET_TRIPLE="x86_64-pc-linux-gnu" \
-DLLVM_TARGETS_TO_BUILD="AMDGPU;NVPTX;Native" \
-DLLVM_ENABLE_PROJECTS="mlir" \
-DLLVM_ENABLE_ASSERTIONS=ON \
-DLLVM_CCACHE_BUILD=ON \
-G Ninja

The find command shows these targets are present, despite what CMake says.

find /root/triton-llvm-build -name "*LLVMNVPTXCodeGen*"

/root/triton-llvm-build/lib/Target/NVPTX/CMakeFiles/LLVMNVPTXCodeGen.dir
/root/triton-llvm-build/lib/libLLVMNVPTXCodeGen.a

find /root/triton-llvm-build -name "*LLVMNVPTXDesc*"

/root/triton-llvm-build/lib/Target/NVPTX/MCTargetDesc/CMakeFiles/LLVMNVPTXDesc.dir
/root/triton-llvm-build/lib/libLLVMNVPTXDesc.a

find /root/triton-llvm-build -name "*LLVMNVPTXInfo*"

/root/triton-llvm-build/lib/Target/NVPTX/TargetInfo/CMakeFiles/LLVMNVPTXInfo.dir
/root/triton-llvm-build/lib/libLLVMNVPTXInfo.a

Environment variables used:

LLVM_LIBRARY_DIR=/root/triton-llvm-build/lib
LLVM_INCLUDE_DIRS=/root/triton-llvm-build/include
LLVM_SYSPATH=/root/triton-llvm-build
pramenku commented 2 weeks ago

https://tritonlang.blob.core.windows.net/llvm-builds/llvm-ed4e505c-ubuntu-x64.tar.gz, this link is working again which was not working earlier.

$ wget https://tritonlang.blob.core.windows.net/llvm-builds/llvm-ed4e505c-ubuntu-x64.tar.gz --2024-08-27 03:25:50-- https://tritonlang.blob.core.windows.net/llvm-builds/llvm-ed4e505c-ubuntu-x64.tar.gz Resolving tritonlang.blob.core.windows.net (tritonlang.blob.core.windows.net)... 20.47.62.100, 20.47.62.112, 20.47.62.60 Connecting to tritonlang.blob.core.windows.net (tritonlang.blob.core.windows.net)|20.47.62.100|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 977588953 (932M) [application/x-tar] Saving to: ‘llvm-ed4e505c-ubuntu-x64.tar.gz’

llvm-ed4e505c-ubuntu-x64.tar.gz 0%[ ] 31.63K 11.9KB/s

TheTrustedComputer commented 2 weeks ago

I can also confirm the links are working again.