search

triton-lang / triton

Development repository for the Triton language and compiler
https://triton-lang.org/
MIT License
13.32k stars 1.63k forks source link

AttributeError in `03-matrix-multiplication.ipynb` tutorial #4014

Open syadegari opened 5 months ago

syadegari commented 5 months ago

Description

Encountered an AttributeError in the 03-matrix-multiplication.ipynb tutorial when attempting to auto-tune configurations for matrix multiplication using Triton.

Error Message

AttributeError: CudaDriver object has no attribute 'active'

Steps to Reproduce

  1. Run the first code cell of the notebook.
  2. The error is triggered when attempting to access triton.runtime.driver.active.get_current_target().

Temporary Fix

I can bypass (not a good fix, but at least runs the notebook and outputs are the same as then ones shown in the online version of tutorial) the error by modifying the device detection and configuration retrieval functions as follows:

def is_cuda():
    return torch.cuda.is_available()

def is_hip_mi200():
    return not is_cuda()

def get_autotune_config():
    if is_cuda():
        return get_cuda_autotune_config()
    else:
        return get_hip_autotune_config()

I appreciate help/tips for a better fix.

lancerts commented 5 months ago 
pip install -U --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/Triton-Nightly/pypi/simple/ triton-nightly

and things should work as is, no need to change the code.

shanguanma commented 5 months ago 
pip install -U --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/Triton-Nightly/pypi/simple/ triton-nightly

and things should work as is, no need to change the code.

I can't install it correct, pip is looking for the correct package, the logger is as follows:

 pip install -U --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/Triton-Nightly/pypi/simple/ triton-nightly
Looking in indexes: https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/Triton-Nightly/pypi/simple/
Collecting triton-nightly
  Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240605045615/triton_nightly-3.0.0.post20240605045615-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (136.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 136.2/136.2 MB 2.3 MB/s eta 0:00:00
Requirement already satisfied: filelock in ./anaconda3/envs/triton-nightly/lib/python3.9/site-packages (from triton-nightly) (3.14.0)
INFO: pip is looking at multiple versions of triton-nightly to determine which version is compatible with other requirements. This could take a while.
  Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240604012636/triton_nightly-3.0.0.post20240604012636-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (136.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 136.0/136.0 MB 3.3 MB/s eta 0:00:00
  Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240603025518/triton_nightly-3.0.0.post20240603025518-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (136.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 136.0/136.0 MB 2.8 MB/s eta 0:00:00
  Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240601044641/triton_nightly-3.0.0.post20240601044641-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (136.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 136.0/136.0 MB 3.0 MB/s eta 0:00:00
  Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240531062509/triton_nightly-3.0.0.post20240531062509-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (136.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 136.0/136.0 MB 2.9 MB/s eta 0:00:00
  Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240530020152/triton_nightly-3.0.0.post20240530020152-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (136.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 136.0/136.0 MB 3.3 MB/s eta 0:00:00
  Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240529030827/triton_nightly-3.0.0.post20240529030827-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (136.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 136.1/136.1 MB 3.3 MB/s eta 0:00:00
  Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240527145827/triton_nightly-3.0.0.post20240527145827-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (135.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 135.7/135.7 MB 3.0 MB/s eta 0:00:00
INFO: pip is still looking at multiple versions of triton-nightly to determine which version is compatible with other requirements. This could take a while.
  Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240526215807/triton_nightly-3.0.0.post20240526215807-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (135.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 135.6/135.6 MB 3.4 MB/s eta 0:00:00
  Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240524232251/triton_nightly-3.0.0.post20240524232251-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (135.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 135.6/135.6 MB 2.8 MB/s eta 0:00:00
  Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240524063540/triton_nightly-3.0.0.post20240524063540-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (135.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 135.6/135.6 MB 3.3 MB/s eta 0:00:00
  Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240522224832/triton_nightly-3.0.0.post20240522224832-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (135.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 135.6/135.6 MB 2.4 MB/s eta 0:00:00
  Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240522054702/triton_nightly-3.0.0.post20240522054702-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (135.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 135.4/135.4 MB 3.3 MB/s eta 0:00:00
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
  Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240521001803/triton_nightly-3.0.0.post20240521001803-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (135.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 135.4/135.4 MB 3.3 MB/s eta 0:00:00
  Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240520020804/triton_nightly-3.0.0.post20240520020804-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (135.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 135.8/135.8 MB 2.2 MB/s eta 0:00:00
  Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240515174524/triton_nightly-3.0.0.post20240515174524-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (135.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 135.2/135.2 MB 3.2 MB/s eta 0:00:00
  Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240513074433/triton_nightly-3.0.0.post20240513074433-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (136.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 136.8/136.8 MB 3.3 MB/s eta 0:00:00
  Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240512011919/triton_nightly-3.0.0.post20240512011919-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (136.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 136.8/136.8 MB 3.3 MB/s eta 0:00:00
  Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240511014100/triton_nightly-3.0.0.post20240511014100-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (136.8 MB)
     ━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 30.1/136.8 MB 3.0 MB/s eta 0:00:36
esteveste commented 5 months ago

I think this is due to the library llnl-hatchet not being available in pip. Probably related to: https://github.com/LLNL/hatchet/issues/134

I did the following to make it work.

pip install git+https://github.com/LLNL/hatchet 

pip install -U --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/Triton-Nightly/pypi/simple/ triton-nightly --use-deprecated legacy-resolver
Jokeren commented 5 months ago

I think the hatchet issue should have been fixed now

esteveste commented 5 months ago

I think the hatchet issue should have been fixed now

Mmm, still does not work out of the box for python 3.11.

$ pip install -U --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/Triton-Nightly/pypi/simple/ triton-nightly --use-deprecated legacy-
resolver                                                                                       
Looking in indexes: https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/Triton-Nightly/pypi/simple/                                                                               Collecting triton-nightly                                                                                                                                                                     
  Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/07c94329-d4c3-4ad4-9e6b-f904a60032ec/pypi/download/triton-nightly/3.post20240610003544/tri
ton_nightly-3.0.0.post20240610003544-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (138.5 MB)                                                                                   
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 138.5/138.5 MB 819.5 kB/s eta 0:00:00                                                                                                           Requirement already satisfied, skipping upgrade: filelock in ./miniconda3/envs/ml-nightly2/lib/python3.11/site-packages (from triton-nightly) (3.13.1)                                        
ERROR: Could not find a version that satisfies the requirement llnl-hatchet (from triton-nightly) (from versions: none)                                                                       
ERROR: No matching distribution found for llnl-hatchet (from triton-nightly)

 $ python --version                                                                                                                                              
Python 3.11.9

But if I do the following it works just fine:

pip install llnl-hatchet
pip install -U --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/Triton-Nightly/pypi/simple/ triton-nightly --use-deprecated legacy-resolver
shanguanma commented 5 months ago

I think this is due to the library llnl-hatchet not being available in pip. Probably related to: LLNL/hatchet#134

I did the following to make it work.

pip install git+https://github.com/LLNL/hatchet 

pip install -U --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/Triton-Nightly/pypi/simple/ triton-nightly --use-deprecated legacy-resolver

Thanks a lot, it works for me. My python environment is python==3.9.