search

PygmalionAI / aphrodite-engine

Large-scale LLM inference engine
https://aphrodite.pygmalion.chat
GNU Affero General Public License v3.0
1.15k stars 127 forks source link

[Installation]: FYI: they fixed the stupid conda pytorch-cuda=12.4 / cuda 12.4.1 strict dependency issue #784

Open BlairSadewitz opened 1 month ago

BlairSadewitz commented 1 month ago

Your current environment

conda now solves this without choking (it never used to work, and I presume that's why you added the nvidia channel at the top):

With this (flexible channel priority, didn't try strict):

name: aphrodite
channels:
  - pytorch
  - nvidia/label/cuda-12.4.1
  - conda-forge
  - defaults
dependencies:
  - python=3.11.*
  - pytorch=2.4.0
  - pytorch-cuda=12.4.*
  - cuda-nvcc=12.4.*
  - cuda-libraries-dev=12.4.*
  - cuda=12.4
  - cuda-version=12.4.*
  - nccl
  - cxx-compiler
  - gxx=11.3
  - pip
  - git=2.35.1
  - ninja

I get:

cuda                      12.4.1                        0    nvidia/label/cuda-12.4.1                                                                                          
cuda-cccl                 12.4.127                      0    nvidia/label/cuda-12.4.1                                                                                          
cuda-command-line-tools   12.4.1                        0    nvidia/label/cuda-12.4.1                                                                                          
cuda-compiler             12.4.1                        0    nvidia/label/cuda-12.4.1                                                                                          
cuda-cudart               12.4.127                      0    nvidia/label/cuda-12.4.1                                                                                          
cuda-cudart-dev           12.4.127                      0    nvidia/label/cuda-12.4.1                                                                                          
cuda-cudart-static        12.4.127                      0    nvidia/label/cuda-12.4.1                                                                                          
cuda-cuobjdump            12.4.127                      0    nvidia/label/cuda-12.4.1                                                                                          
cuda-cupti                12.4.127                      0    nvidia/label/cuda-12.4.1                                                                                          
cuda-cupti-static         12.4.127                      0    nvidia/label/cuda-12.4.1                                                                                          
cuda-cuxxfilt             12.4.127                      0    nvidia/label/cuda-12.4.1                                                                                          
cuda-demo-suite           12.4.127                      0    nvidia/label/cuda-12.4.1                                                                                          
cuda-documentation        12.4.127                      0    nvidia/label/cuda-12.4.1                                                                                          
cuda-driver-dev           12.4.127                      0    nvidia/label/cuda-12.4.1                                                                                          
cuda-gdb                  12.4.127                      0    nvidia/label/cuda-12.4.1                                                                                          
cuda-libraries            12.4.1                        0    nvidia/label/cuda-12.4.1                                                                                          
cuda-libraries-dev        12.4.1                        0    nvidia/label/cuda-12.4.1  
cuda-libraries-static     12.4.1                        0    nvidia/label/cuda-12.4.1  
cuda-nsight               12.4.127                      0    nvidia/label/cuda-12.4.1                                                                                          
cuda-nsight-compute       12.4.1                        0    nvidia/label/cuda-12.4.1                                                                                          
cuda-nvcc                 12.4.131                      0    nvidia/label/cuda-12.4.1                                                                                          
cuda-nvdisasm             12.4.127                      0    nvidia/label/cuda-12.4.1                                                                                          
cuda-nvml-dev             12.4.127                      0    nvidia/label/cuda-12.4.1                                                                                          
cuda-nvprof               12.4.127                      0    nvidia/label/cuda-12.4.1                                                                                          
cuda-nvprune              12.4.127                      0    nvidia/label/cuda-12.4.1                                                                                          
cuda-nvrtc                12.4.127                      0    nvidia/label/cuda-12.4.1                                                                                          
cuda-nvrtc-dev            12.4.127                      0    nvidia/label/cuda-12.4.1                                                                                          
cuda-nvrtc-static         12.4.127                      0    nvidia/label/cuda-12.4.1                                                                                          
cuda-nvtx                 12.4.127                      0    nvidia/label/cuda-12.4.1                                                                                          
cuda-nvvp                 12.4.127                      0    nvidia/label/cuda-12.4.1                                                                                          
cuda-opencl               12.4.127                      0    nvidia/label/cuda-12.4.1                                                                                          
cuda-opencl-dev           12.4.127                      0    nvidia/label/cuda-12.4.1                                                                                          
cuda-profiler-api         12.4.127                      0    nvidia/label/cuda-12.4.1                                                                                          
cuda-runtime              12.4.1                        0    nvidia/label/cuda-12.4.1                                                                                          
cuda-sanitizer-api        12.4.127                      0    nvidia/label/cuda-12.4.1                                                                                          
cuda-toolkit              12.4.1                        0    nvidia/label/cuda-12.4.1                                                                                          
cuda-tools                12.4.1                        0    nvidia/label/cuda-12.4.1                                                                                          
cuda-version              12.4                 h3060b56_3    conda-forge                                                                                                       
cuda-visual-tools         12.4.1                        0    nvidia/label/cuda-12.4.1                                                                                          
gds-tools                 1.9.1.3                       0    nvidia/label/cuda-12.4.1                                                                                          
libcublas                 12.4.5.8                      0    nvidia/label/cuda-12.4.1                                                                                          
libcublas-dev             12.4.5.8                      0    nvidia/label/cuda-12.4.1                                                                                          
libcublas-static          12.4.5.8                      0    nvidia/label/cuda-12.4.1                                                                                          
libcufft                  11.2.1.3                      0    nvidia/label/cuda-12.4.1                                                                                          
libcufft-dev              11.2.1.3                      0    nvidia/label/cuda-12.4.1                                                                                          
libcufft-static           11.2.1.3                      0    nvidia/label/cuda-12.4.1                                                                                          
libcufile                 1.9.1.3                       0    nvidia/label/cuda-12.4.1                                                                                          
libcufile-dev             1.9.1.3                       0    nvidia/label/cuda-12.4.1                                                                                          
libcufile-static          1.9.1.3                       0    nvidia/label/cuda-12.4.1                                                                                          
libcurand                 10.3.5.147                    0    nvidia/label/cuda-12.4.1                                                                                          
libcurand-dev             10.3.5.147                    0    nvidia/label/cuda-12.4.1                                                                                          
libcurand-static          10.3.5.147                    0    nvidia/label/cuda-12.4.1                                                                                          
libcusolver               11.6.1.9                      0    nvidia/label/cuda-12.4.1                                                                                          
libcusolver-dev           11.6.1.9                      0    nvidia/label/cuda-12.4.1                                                                                          
libcusolver-static        11.6.1.9                      0    nvidia/label/cuda-12.4.1                                                                                          
libcusparse               12.3.1.170                    0    nvidia/label/cuda-12.4.1                                                                                          
libcusparse-dev           12.3.1.170                    0    nvidia/label/cuda-12.4.1                                                                                          
libcusparse-static        12.3.1.170                    0    nvidia/label/cuda-12.4.1                                                                                          
libnpp                    12.2.5.30                     0    nvidia/label/cuda-12.4.1                                                                                          
libnpp-dev                12.2.5.30                     0    nvidia/label/cuda-12.4.1                                                                                          
libnpp-static             12.2.5.30                     0    nvidia/label/cuda-12.4.1                                                                                          
libnvfatbin               12.4.127                      0    nvidia/label/cuda-12.4.1                                                                                          
libnvfatbin-dev           12.4.127                      0    nvidia/label/cuda-12.4.1                                                                                          
libnvjitlink              12.4.127                      0    nvidia/label/cuda-12.4.1                                                                                          
libnvjitlink-dev          12.4.127                      0    nvidia/label/cuda-12.4.1                                                                                          
libnvjpeg                 12.3.1.117                    0    nvidia/label/cuda-12.4.1                                                                                          
libnvjpeg-dev             12.3.1.117                    0    nvidia/label/cuda-12.4.1                                                                                          
libnvjpeg-static          12.3.1.117                    0    nvidia/label/cuda-12.4.1                                                                                          
nsight-compute            2024.1.1.4                    0    nvidia/label/cuda-12.4.1                                                                                          
pytorch                   2.4.0           py3.11_cuda12.4_cudnn9.1.0_0    pytorch                                                                                              
pytorch-cuda              12.4                 hc786d27_7    pytorch                                                  
pytorch-mutex             1.0                        cuda    pytorch

Ain't life grand?

Collecting environment information...
PyTorch version: 2.4.0
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.4 LTS (x86_64)
GCC version: (conda-forge gcc 11.3.0-19) 11.3.0
Clang version: Could not collect
CMake version: version 3.26.4
Libc version: glibc-2.35

Python version: 3.11.9 | packaged by conda-forge | (main, Apr 19 2024, 18:36:13) [GCC 12.3.0] (64-bit runtime)
Python platform: Linux-6.8.0-45-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.4.131
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA H100 PCIe
GPU 1: NVIDIA H100 PCIe

Nvidia driver version: 560.35.03
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        52 bits physical, 57 bits virtual
Byte Order:                           Little Endian
CPU(s):                               256
On-line CPU(s) list:                  0-255
Vendor ID:                            AuthenticAMD
Model name:                           AMD EPYC 9554 64-Core Processor
CPU family:                           25
Model:                                17
Thread(s) per core:                   2
Core(s) per socket:                   64
Socket(s):                            2
Stepping:                             1
Frequency boost:                      enabled
CPU max MHz:                          3762.9880
CPU min MHz:                          1500.0000
BogoMIPS:                             6191.12
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d debug_swap
Virtualization:                       AMD-V
L1d cache:                            4 MiB (128 instances)
L1i cache:                            4 MiB (128 instances)
L2 cache:                             128 MiB (128 instances)
L3 cache:                             512 MiB (16 instances)
NUMA node(s):                         2
NUMA node0 CPU(s):                    0-63,128-191
NUMA node1 CPU(s):                    64-127,192-255
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Mitigation; Safe RET
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

Versions of relevant libraries:
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

Versions of relevant libraries:
[pip3] torch==2.4.0
[pip3] triton==3.0.0
[conda] blas                      2.16                        mkl    conda-forge
[conda] libblas                   3.8.0                    16_mkl    conda-forge
[conda] libcblas                  3.8.0                    16_mkl    conda-forge
[conda] liblapack                 3.8.0                    16_mkl    conda-forge
[conda] liblapacke                3.8.0                    16_mkl    conda-forge
[conda] mkl                       2020.2                      256  
[conda] nccl                      2.23.4.1             h52f6c39_0    conda-forge
[conda] pytorch                   2.4.0           py3.11_cuda12.4_cudnn9.1.0_0    pytorch
[conda] pytorch-cuda              12.4                 hc786d27_7    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchtriton               3.0.0                     py311    pytorch
ROCM Version: Could not collect
Neuron SDK Version: N/A
Aphrodite Version: N/A
Aphrodite Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
ESC[4mGPU0      GPU1    CPU Affinity    NUMA Affinity   GPU NUMA IDESC[0m
GPU0     X      NODE    64-127,192-255  1               N/A
GPU1    NODE     X      64-127,192-255  1               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

How did you install Aphrodite?

N/A