search

vllm-project / vllm

A high-throughput and memory-efficient inference and serving engine for LLMs
https://docs.vllm.ai
Apache License 2.0
30.05k stars 4.54k forks source link

ImportError: /ramyapra/vllm/vllm/_C.cpython-310-x86_64-linux-gnu.so: undefined symbol: #2747

Closed ramyaprabhu-alt closed 7 months ago

ramyaprabhu-alt commented 9 months ago

I'm trying to run vllm and lm-eval-harness. I'm using vllm 0.2.5. After I'm done installing both, if I try importing vllm I get the following error: File "/ramyapra/lm-evaluation-harness/lm_eval/models/__init__.py", line 7, in <module> from . import vllm_causallms File "/ramyapra/lm-evaluation-harness/lm_eval/models/vllm_causallms.py", line 16, in <module> from vllm import LLM, SamplingParams File "/ramyapra/vllm/vllm/__init__.py", line 3, in <module> from vllm.engine.arg_utils import AsyncEngineArgs, EngineArgs File "/ramyapra/vllm/vllm/engine/arg_utils.py", line 6, in <module> from vllm.config import (CacheConfig, ModelConfig, ParallelConfig, File "/ramyapra/vllm/vllm/config.py", line 9, in <module> from vllm.utils import get_cpu_memory, is_hip File "/ramyapra/vllm/vllm/utils.py", line 8, in <module> from vllm._C import cuda_utils ImportError: /ramyapra/vllm/vllm/_C.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN2at4_ops19empty_memory_format4callEN3c108ArrayRefINS2_6SymIntEEESt8optionalINS2_10ScalarTypeEES6_INS2_6LayoutEES6_INS2_6DeviceEES6_IbES6_INS2_12MemoryFormatEE

I'm using the NGC docker container 23:10-py3.

sudarshan-kamath commented 9 months ago

Uninstall the package called transformer-engine by using the command pip uninstall transformer-engine. Install transformers and accelerate if not already installed. Then it should work.

bfotzo commented 9 months ago

Uninstall the package called transformer-engine by using the command pip uninstall transformer-engine. Install transformers and accelerate if not already installed. Then it should work.

I tried this but didn't work

sudarshan-kamath commented 9 months ago

Please post the vllm version and then the steps to reproduce this. Which model are you using? Cuda version on the system and the docker (if you use it).

hassan-twelvelabs commented 9 months ago

I am also experiencing this issue. env: vllm-0.2.2 , cuda 12.2 , no docker. Local build of vllm from source, then running examples/offline_inference.py.

choco9966 commented 8 months ago

i also experience vllm==0.2.5 / accelerate==0.27.2 / transformers==4.37.2

sylcjl commented 8 months ago

Any idea? I am also facing this issue Docker image: NGC docker container 23.10-py3 too.

rkuo2000 commented 8 months ago

same problem, I am running on Kaggle.com

Successfully installed aioprometheus-23.12.0 cupy-cuda12x-12.1.0 pynvml-11.5.0 quantile-python-1.1 transformers-4.38.1 triton-2.2.0 vllm-0.3.2 xformers-0.0.23.post1

sylcjl commented 8 months ago

It seems to be PyTorch isn't working with Cuda-12.2 on NGC-PyTorch 23.10-py3. I fix this issue on NGC-PyTorch 23.07-py3 and reinstall pytorch with v2.1.2

mohammad-yousuf commented 8 months ago

same issue. anyone able to fix?

cuda: 12.0.1 torch : 2.2.1 transformers: 4.38.2 vllm: 0.3.2 accelerate: 0.22.0

UPDATE: solved by downgrading torch to 2.1.2

surak commented 8 months ago

2797 is the same

RylanSchaeffer commented 7 months ago

I'm hitting this same problem :(

RylanSchaeffer commented 7 months ago

@sudarshan-kamath , pip uninstall transformer-engine also did not work for me

youkaichao commented 7 months ago

@RylanSchaeffer can you try to installing a matching version of pytorch with vllm?

e.g. vllm 0.3.3 with pytorch 2.1 .

RylanSchaeffer commented 7 months ago

Versions:

>>> import torch
>>> torch.__version__
'2.1.2.post301'
>>> import vllm
>>> vllm.__version__
'0.3.3'

Error:

ImportError: /lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/lib/python3.10/site-packages/vllm/_C.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c106detail23torchInternalAssertFailEPKcS2_jS2_RKSs
RylanSchaeffer commented 7 months ago

@youkaichao , here's a script I'm using to debug:

from vllm import LLM, SamplingParams

sampling_params = SamplingParams(max_tokens=50, temperature=1.0)

llm = LLM(
    model=f"gpt2",
    tensor_parallel_size=4,
    # load_format="pt",
)

outputs = llm.generate(
    ["My name is Lewis and I like to", "Today is a good day to", "John has"],
    sampling_params,
)

for output in outputs:
    print(output.outputs[0].text)
    print("=========")
youkaichao commented 7 months ago

2.1.2.post301

Are you using a custom built version of pytorch?

vLLM is compiled against officially released pytorch. And there is no binary compatibility promise across pytorch versions.

You can try to build vllm yourself: https://docs.vllm.ai/en/latest/getting_started/installation.html#build-from-source .

RylanSchaeffer commented 7 months ago

Are you using a custom built version of pytorch?

No, I installed using the default command from pytorch itself (conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia)

I am now trying the following: I deleted my conda environment, added vllm as a git submodules and am now trying pip install -e .

This is the error I received:

Building wheels for collected packages: vllm
  Building editable for vllm (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building editable for vllm (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [145 lines of output]
      /tmp/user/21163/pip-build-env-a245et9d/overlay/lib/python3.10/site-packages/torch/nn/modules/transformer.py:20: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at ../torch/csrc/utils/tensor_numpy.cpp:84.)
        device: torch.device = torch.device(torch._C._get_default_device()),  # torch.device('cpu'),
      running editable_wheel
      creating /tmp/user/21163/pip-wheel-zf61weqe/.tmp-id2wfdg6/vllm.egg-info
      writing /tmp/user/21163/pip-wheel-zf61weqe/.tmp-id2wfdg6/vllm.egg-info/PKG-INFO
      writing dependency_links to /tmp/user/21163/pip-wheel-zf61weqe/.tmp-id2wfdg6/vllm.egg-info/dependency_links.txt
      writing requirements to /tmp/user/21163/pip-wheel-zf61weqe/.tmp-id2wfdg6/vllm.egg-info/requires.txt
      writing top-level names to /tmp/user/21163/pip-wheel-zf61weqe/.tmp-id2wfdg6/vllm.egg-info/top_level.txt
      writing manifest file '/tmp/user/21163/pip-wheel-zf61weqe/.tmp-id2wfdg6/vllm.egg-info/SOURCES.txt'
      reading manifest file '/tmp/user/21163/pip-wheel-zf61weqe/.tmp-id2wfdg6/vllm.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      adding license file 'LICENSE'
      writing manifest file '/tmp/user/21163/pip-wheel-zf61weqe/.tmp-id2wfdg6/vllm.egg-info/SOURCES.txt'
      creating '/tmp/user/21163/pip-wheel-zf61weqe/.tmp-id2wfdg6/vllm-0.3.3+cu122.dist-info'
      creating /tmp/user/21163/pip-wheel-zf61weqe/.tmp-id2wfdg6/vllm-0.3.3+cu122.dist-info/WHEEL
      running build_py
      running build_ext
      -- The CXX compiler identification is GNU 9.4.0
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Build type: RelWithDebInfo
      -- Found Python: /lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/bin/python3.10 (found version "3.10.14") found components: Interpreter Development.Module
      -- Found python matching: /lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/bin/python3.10.
      -- Found CUDA: /usr/local/cuda (found version "12.2")
      -- The CUDA compiler identification is unknown
      CMake Error at /tmp/user/21163/pip-build-env-a245et9d/overlay/lib/python3.10/site-packages/cmake/data/share/cmake-3.28/Modules/CMakeDetermineCUDACompiler.cmake:270 (message):
        Failed to detect a default CUDA architecture.

        Compiler output:

      Call Stack (most recent call first):
        /tmp/user/21163/pip-build-env-a245et9d/overlay/lib/python3.10/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:47 (enable_language)
        /tmp/user/21163/pip-build-env-a245et9d/overlay/lib/python3.10/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:87 (include)
        /tmp/user/21163/pip-build-env-a245et9d/overlay/lib/python3.10/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
        CMakeLists.txt:64 (find_package)

      -- Configuring incomplete, errors occurred!
      Traceback (most recent call last):
        File "/tmp/user/21163/pip-build-env-a245et9d/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 155, in run
          self._create_wheel_file(bdist_wheel)
        File "/tmp/user/21163/pip-build-env-a245et9d/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 357, in _create_wheel_file
          files, mapping = self._run_build_commands(dist_name, unpacked, lib, tmp)
        File "/tmp/user/21163/pip-build-env-a245et9d/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 280, in _run_build_commands
          self._run_build_subcommands()
        File "/tmp/user/21163/pip-build-env-a245et9d/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 307, in _run_build_subcommands
          self.run_command(name)
        File "/tmp/user/21163/pip-build-env-a245et9d/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/tmp/user/21163/pip-build-env-a245et9d/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 967, in run_command
          super().run_command(command)
        File "/tmp/user/21163/pip-build-env-a245et9d/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/user/21163/pip-build-env-a245et9d/overlay/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 91, in run
          _build_ext.run(self)
        File "/tmp/user/21163/pip-build-env-a245et9d/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
          self.build_extensions()
        File "<string>", line 164, in build_extensions
        File "<string>", line 147, in configure
        File "/lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/lib/python3.10/subprocess.py", line 369, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', '/lfs/ampere8/0/rschaef/KoyejoLab-Revisiting-Model-Collapse/submodules/vllm', '-G', 'Ninja', '-DCMAKE_BUILD_TYPE=RelWithDebInfo', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/tmp/user/21163/tmproow4us4.build-lib/vllm', '-DCMAKE_ARCHIVE_OUTPUT_DIRECTORY=/tmp/user/21163/tmpqy5i3vwd.build-temp', '-DVLLM_PYTHON_EXECUTABLE=/lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/bin/python3.10', '-DNVCC_THREADS=8', '-DCMAKE_JOB_POOL_COMPILE:STRING=compile', '-DCMAKE_JOB_POOLS:STRING=compile=64']' returned non-zero exit status 1.
      /tmp/user/21163/pip-build-env-a245et9d/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py:988: _DebuggingTips: Problem in editable installation.
      !!

              ********************************************************************************
              An error happened while installing `vllm` in editable mode.

              The following steps are recommended to help debug this problem:

              - Try to install the project normally, without using the editable mode.
                Does the error still persist?
                (If it does, try fixing the problem before attempting the editable mode).
              - If you are using binary extensions, make sure you have all OS-level
                dependencies installed (e.g. compilers, toolchains, binary libraries, ...).
              - Try the latest version of setuptools (maybe the error was already fixed).
              - If you (or your project dependencies) are using any setuptools extension
                or customization, make sure they support the editable mode.

              After following the steps above, if the problem still persists and
              you think this is related to how setuptools handles editable installations,
              please submit a reproducible example
              (see https://stackoverflow.com/help/minimal-reproducible-example) to:

                  https://github.com/pypa/setuptools/issues

              See https://setuptools.pypa.io/en/latest/userguide/development_mode.html for details.
              ********************************************************************************

      !!
        cmd_obj.run()
      Traceback (most recent call last):
        File "/lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 273, in build_editable
          return hook(wheel_directory, config_settings, metadata_directory)
        File "/tmp/user/21163/pip-build-env-a245et9d/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 443, in build_editable
          return self._build_with_temp_dir(
        File "/tmp/user/21163/pip-build-env-a245et9d/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 395, in _build_with_temp_dir
          self.run_setup()
        File "/tmp/user/21163/pip-build-env-a245et9d/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 345, in <module>
        File "/tmp/user/21163/pip-build-env-a245et9d/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 104, in setup
          return distutils.core.setup(**attrs)
        File "/tmp/user/21163/pip-build-env-a245et9d/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
        File "/tmp/user/21163/pip-build-env-a245et9d/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/tmp/user/21163/pip-build-env-a245et9d/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/user/21163/pip-build-env-a245et9d/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 967, in run_command
          super().run_command(command)
        File "/tmp/user/21163/pip-build-env-a245et9d/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/user/21163/pip-build-env-a245et9d/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 155, in run
          self._create_wheel_file(bdist_wheel)
        File "/tmp/user/21163/pip-build-env-a245et9d/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 357, in _create_wheel_file
          files, mapping = self._run_build_commands(dist_name, unpacked, lib, tmp)
        File "/tmp/user/21163/pip-build-env-a245et9d/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 280, in _run_build_commands
          self._run_build_subcommands()
        File "/tmp/user/21163/pip-build-env-a245et9d/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 307, in _run_build_subcommands
          self.run_command(name)
        File "/tmp/user/21163/pip-build-env-a245et9d/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/tmp/user/21163/pip-build-env-a245et9d/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 967, in run_command
          super().run_command(command)
        File "/tmp/user/21163/pip-build-env-a245et9d/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/user/21163/pip-build-env-a245et9d/overlay/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 91, in run
          _build_ext.run(self)
        File "/tmp/user/21163/pip-build-env-a245et9d/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
          self.build_extensions()
        File "<string>", line 164, in build_extensions
        File "<string>", line 147, in configure
        File "/lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/lib/python3.10/subprocess.py", line 369, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', '/lfs/ampere8/0/rschaef/KoyejoLab-Revisiting-Model-Collapse/submodules/vllm', '-G', 'Ninja', '-DCMAKE_BUILD_TYPE=RelWithDebInfo', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/tmp/user/21163/tmproow4us4.build-lib/vllm', '-DCMAKE_ARCHIVE_OUTPUT_DIRECTORY=/tmp/user/21163/tmpqy5i3vwd.build-temp', '-DVLLM_PYTHON_EXECUTABLE=/lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/bin/python3.10', '-DNVCC_THREADS=8', '-DCMAKE_JOB_POOL_COMPILE:STRING=compile', '-DCMAKE_JOB_POOLS:STRING=compile=64']' returned non-zero exit status 1.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building editable for vllm
Failed to build vllm
ERROR: Could not build wheels for vllm, which is required to install pyproject.toml-based projects
RylanSchaeffer commented 7 months ago

I'm deleting my pip and conda cache and trying again

RylanSchaeffer commented 7 months ago

I purged my .cache/pip and .cache/conda. Trying pip install -e . yielded the same error:

Building wheels for collected packages: vllm
  Building editable for vllm (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building editable for vllm (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [145 lines of output]
      /tmp/user/21163/pip-build-env-ygusfc74/overlay/lib/python3.10/site-packages/torch/nn/modules/transformer.py:20: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at ../torch/csrc/utils/tensor_numpy.cpp:84.)
        device: torch.device = torch.device(torch._C._get_default_device()),  # torch.device('cpu'),
      running editable_wheel
      creating /tmp/user/21163/pip-wheel-6kdhr3i2/.tmp-p2xpmenf/vllm.egg-info
      writing /tmp/user/21163/pip-wheel-6kdhr3i2/.tmp-p2xpmenf/vllm.egg-info/PKG-INFO
      writing dependency_links to /tmp/user/21163/pip-wheel-6kdhr3i2/.tmp-p2xpmenf/vllm.egg-info/dependency_links.txt
      writing requirements to /tmp/user/21163/pip-wheel-6kdhr3i2/.tmp-p2xpmenf/vllm.egg-info/requires.txt
      writing top-level names to /tmp/user/21163/pip-wheel-6kdhr3i2/.tmp-p2xpmenf/vllm.egg-info/top_level.txt
      writing manifest file '/tmp/user/21163/pip-wheel-6kdhr3i2/.tmp-p2xpmenf/vllm.egg-info/SOURCES.txt'
      reading manifest file '/tmp/user/21163/pip-wheel-6kdhr3i2/.tmp-p2xpmenf/vllm.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      adding license file 'LICENSE'
      writing manifest file '/tmp/user/21163/pip-wheel-6kdhr3i2/.tmp-p2xpmenf/vllm.egg-info/SOURCES.txt'
      creating '/tmp/user/21163/pip-wheel-6kdhr3i2/.tmp-p2xpmenf/vllm-0.3.3+cu122.dist-info'
      creating /tmp/user/21163/pip-wheel-6kdhr3i2/.tmp-p2xpmenf/vllm-0.3.3+cu122.dist-info/WHEEL
      running build_py
      running build_ext
      -- The CXX compiler identification is GNU 9.4.0
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Build type: RelWithDebInfo
      -- Found Python: /lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/bin/python3.10 (found version "3.10.14") found components: Interpreter Development.Module
      -- Found python matching: /lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/bin/python3.10.
      -- Found CUDA: /usr/local/cuda (found version "12.2")
      -- The CUDA compiler identification is unknown
      CMake Error at /tmp/user/21163/pip-build-env-ygusfc74/overlay/lib/python3.10/site-packages/cmake/data/share/cmake-3.28/Modules/CMakeDetermineCUDACompiler.cmake:270 (message):
        Failed to detect a default CUDA architecture.

        Compiler output:

      Call Stack (most recent call first):
        /tmp/user/21163/pip-build-env-ygusfc74/overlay/lib/python3.10/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:47 (enable_language)
        /tmp/user/21163/pip-build-env-ygusfc74/overlay/lib/python3.10/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:87 (include)
        /tmp/user/21163/pip-build-env-ygusfc74/overlay/lib/python3.10/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
        CMakeLists.txt:64 (find_package)

      -- Configuring incomplete, errors occurred!
      Traceback (most recent call last):
        File "/tmp/user/21163/pip-build-env-ygusfc74/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 155, in run
          self._create_wheel_file(bdist_wheel)
        File "/tmp/user/21163/pip-build-env-ygusfc74/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 357, in _create_wheel_file
          files, mapping = self._run_build_commands(dist_name, unpacked, lib, tmp)
        File "/tmp/user/21163/pip-build-env-ygusfc74/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 280, in _run_build_commands
          self._run_build_subcommands()
        File "/tmp/user/21163/pip-build-env-ygusfc74/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 307, in _run_build_subcommands
          self.run_command(name)
        File "/tmp/user/21163/pip-build-env-ygusfc74/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/tmp/user/21163/pip-build-env-ygusfc74/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 967, in run_command
          super().run_command(command)
        File "/tmp/user/21163/pip-build-env-ygusfc74/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/user/21163/pip-build-env-ygusfc74/overlay/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 91, in run
          _build_ext.run(self)
        File "/tmp/user/21163/pip-build-env-ygusfc74/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
          self.build_extensions()
        File "<string>", line 164, in build_extensions
        File "<string>", line 147, in configure
        File "/lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/lib/python3.10/subprocess.py", line 369, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', '/lfs/ampere8/0/rschaef/KoyejoLab-Revisiting-Model-Collapse/submodules/vllm', '-G', 'Ninja', '-DCMAKE_BUILD_TYPE=RelWithDebInfo', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/tmp/user/21163/tmp66kpgsiv.build-lib/vllm', '-DCMAKE_ARCHIVE_OUTPUT_DIRECTORY=/tmp/user/21163/tmpzvgnump_.build-temp', '-DVLLM_PYTHON_EXECUTABLE=/lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/bin/python3.10', '-DNVCC_THREADS=8', '-DCMAKE_JOB_POOL_COMPILE:STRING=compile', '-DCMAKE_JOB_POOLS:STRING=compile=64']' returned non-zero exit status 1.
      /tmp/user/21163/pip-build-env-ygusfc74/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py:988: _DebuggingTips: Problem in editable installation.
      !!

              ********************************************************************************
              An error happened while installing `vllm` in editable mode.

              The following steps are recommended to help debug this problem:

              - Try to install the project normally, without using the editable mode.
                Does the error still persist?
                (If it does, try fixing the problem before attempting the editable mode).
              - If you are using binary extensions, make sure you have all OS-level
                dependencies installed (e.g. compilers, toolchains, binary libraries, ...).
              - Try the latest version of setuptools (maybe the error was already fixed).
              - If you (or your project dependencies) are using any setuptools extension
                or customization, make sure they support the editable mode.

              After following the steps above, if the problem still persists and
              you think this is related to how setuptools handles editable installations,
              please submit a reproducible example
              (see https://stackoverflow.com/help/minimal-reproducible-example) to:

                  https://github.com/pypa/setuptools/issues

              See https://setuptools.pypa.io/en/latest/userguide/development_mode.html for details.
              ********************************************************************************

      !!
        cmd_obj.run()
      Traceback (most recent call last):
        File "/lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 273, in build_editable
          return hook(wheel_directory, config_settings, metadata_directory)
        File "/tmp/user/21163/pip-build-env-ygusfc74/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 443, in build_editable
          return self._build_with_temp_dir(
        File "/tmp/user/21163/pip-build-env-ygusfc74/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 395, in _build_with_temp_dir
          self.run_setup()
        File "/tmp/user/21163/pip-build-env-ygusfc74/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 345, in <module>
        File "/tmp/user/21163/pip-build-env-ygusfc74/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 104, in setup
          return distutils.core.setup(**attrs)
        File "/tmp/user/21163/pip-build-env-ygusfc74/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
        File "/tmp/user/21163/pip-build-env-ygusfc74/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/tmp/user/21163/pip-build-env-ygusfc74/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/user/21163/pip-build-env-ygusfc74/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 967, in run_command
          super().run_command(command)
        File "/tmp/user/21163/pip-build-env-ygusfc74/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/user/21163/pip-build-env-ygusfc74/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 155, in run
          self._create_wheel_file(bdist_wheel)
        File "/tmp/user/21163/pip-build-env-ygusfc74/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 357, in _create_wheel_file
          files, mapping = self._run_build_commands(dist_name, unpacked, lib, tmp)
        File "/tmp/user/21163/pip-build-env-ygusfc74/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 280, in _run_build_commands
          self._run_build_subcommands()
        File "/tmp/user/21163/pip-build-env-ygusfc74/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 307, in _run_build_subcommands
          self.run_command(name)
        File "/tmp/user/21163/pip-build-env-ygusfc74/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/tmp/user/21163/pip-build-env-ygusfc74/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 967, in run_command
          super().run_command(command)
        File "/tmp/user/21163/pip-build-env-ygusfc74/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/user/21163/pip-build-env-ygusfc74/overlay/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 91, in run
          _build_ext.run(self)
        File "/tmp/user/21163/pip-build-env-ygusfc74/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
          self.build_extensions()
        File "<string>", line 164, in build_extensions
        File "<string>", line 147, in configure
        File "/lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/lib/python3.10/subprocess.py", line 369, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', '/lfs/ampere8/0/rschaef/KoyejoLab-Revisiting-Model-Collapse/submodules/vllm', '-G', 'Ninja', '-DCMAKE_BUILD_TYPE=RelWithDebInfo', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/tmp/user/21163/tmp66kpgsiv.build-lib/vllm', '-DCMAKE_ARCHIVE_OUTPUT_DIRECTORY=/tmp/user/21163/tmpzvgnump_.build-temp', '-DVLLM_PYTHON_EXECUTABLE=/lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/bin/python3.10', '-DNVCC_THREADS=8', '-DCMAKE_JOB_POOL_COMPILE:STRING=compile', '-DCMAKE_JOB_POOLS:STRING=compile=64']' returned non-zero exit status 1.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building editable for vllm
Failed to build vllm
ERROR: Could not build wheels for vllm, which is required to install pyproject.toml-based projects
youkaichao commented 7 months ago

Failed to detect a default CUDA architecture.

You don't have a valid cuda installation. Try which nvcc , and you can see it will fail.

You can install one by conda install cuda=12.1 -c nvidia -y .

RylanSchaeffer commented 7 months ago

@youkaichao thanks for help! New error:

Building wheels for collected packages: vllm
  Building editable for vllm (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building editable for vllm (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [175 lines of output]
      /tmp/user/21163/pip-build-env-4trpsjo5/overlay/lib/python3.10/site-packages/torch/nn/modules/transformer.py:20: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at ../torch/csrc/utils/tensor_numpy.cpp:84.)
        device: torch.device = torch.device(torch._C._get_default_device()),  # torch.device('cpu'),
      running editable_wheel
      creating /tmp/user/21163/pip-wheel-_q87sc5u/.tmp-g7p5fhai/vllm.egg-info
      writing /tmp/user/21163/pip-wheel-_q87sc5u/.tmp-g7p5fhai/vllm.egg-info/PKG-INFO
      writing dependency_links to /tmp/user/21163/pip-wheel-_q87sc5u/.tmp-g7p5fhai/vllm.egg-info/dependency_links.txt
      writing requirements to /tmp/user/21163/pip-wheel-_q87sc5u/.tmp-g7p5fhai/vllm.egg-info/requires.txt
      writing top-level names to /tmp/user/21163/pip-wheel-_q87sc5u/.tmp-g7p5fhai/vllm.egg-info/top_level.txt
      writing manifest file '/tmp/user/21163/pip-wheel-_q87sc5u/.tmp-g7p5fhai/vllm.egg-info/SOURCES.txt'
      reading manifest file '/tmp/user/21163/pip-wheel-_q87sc5u/.tmp-g7p5fhai/vllm.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      adding license file 'LICENSE'
      writing manifest file '/tmp/user/21163/pip-wheel-_q87sc5u/.tmp-g7p5fhai/vllm.egg-info/SOURCES.txt'
      creating '/tmp/user/21163/pip-wheel-_q87sc5u/.tmp-g7p5fhai/vllm-0.3.3+cu124.dist-info'
      creating /tmp/user/21163/pip-wheel-_q87sc5u/.tmp-g7p5fhai/vllm-0.3.3+cu124.dist-info/WHEEL
      running build_py
      running build_ext
      -- The CXX compiler identification is GNU 9.4.0
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Build type: RelWithDebInfo
      -- Found Python: /lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/bin/python3.10 (found version "3.10.14") found components: Interpreter Development.Module
      -- Found python matching: /lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/bin/python3.10.
      -- Found CUDA: /lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env (found version "12.4")
      -- The CUDA compiler identification is NVIDIA 12.4.99
      -- Detecting CUDA compiler ABI info
      -- Detecting CUDA compiler ABI info - done
      -- Check for working CUDA compiler: /lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/bin/nvcc - skipped
      -- Detecting CUDA compile features
      -- Detecting CUDA compile features - done
      -- Found CUDAToolkit: /lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/include (found version "12.4.99")
      -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
      -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
      -- Looking for pthread_create in pthreads
      -- Looking for pthread_create in pthreads - not found
      -- Looking for pthread_create in pthread
      -- Looking for pthread_create in pthread - found
      -- Found Threads: TRUE
      -- Caffe2: CUDA detected: 12.4
      -- Caffe2: CUDA nvcc is: /lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/bin/nvcc
      -- Caffe2: CUDA toolkit directory: /lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env
      -- Caffe2: Header version is: 12.4
      -- /lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/lib/libnvrtc.so shorthash is a438b617
      -- USE_CUDNN is set to 0. Compiling without cuDNN support
      -- USE_CUSPARSELT is set to 0. Compiling without cuSPARSELt support
      -- Autodetected CUDA architecture(s):  8.0 8.0 8.0 8.0 8.0 8.0 8.0 8.0
      -- Added CUDA NVCC flags for: -gencode;arch=compute_80,code=sm_80
      CMake Warning at /tmp/user/21163/pip-build-env-4trpsjo5/overlay/lib/python3.10/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:22 (message):
        static library kineto_LIBRARY-NOTFOUND not found.
      Call Stack (most recent call first):
        /tmp/user/21163/pip-build-env-4trpsjo5/overlay/lib/python3.10/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:127 (append_torchlib_if_found)
        CMakeLists.txt:64 (find_package)

      -- Found Torch: /tmp/user/21163/pip-build-env-4trpsjo5/overlay/lib/python3.10/site-packages/torch/lib/libtorch.so
      -- CUDA supported arches: 7.0;7.5;8.0;8.6;8.9;9.0
      -- CUDA target arches: 80
      -- Punica target arches: 80
      -- Enabling C extension.
      -- Enabling moe extension.
      -- Configuring done (6.6s)
      CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
      Please set them or make sure they are set and tested correctly in the CMake files:
      LIBNVTOOLSEXT
          linked by target "_punica_C" in directory /lfs/ampere8/0/rschaef/KoyejoLab-Revisiting-Model-Collapse/submodules/vllm
          linked by target "_moe_C" in directory /lfs/ampere8/0/rschaef/KoyejoLab-Revisiting-Model-Collapse/submodules/vllm
          linked by target "_C" in directory /lfs/ampere8/0/rschaef/KoyejoLab-Revisiting-Model-Collapse/submodules/vllm

      -- Generating done (0.0s)
      CMake Generate step failed.  Build files cannot be regenerated correctly.
      Traceback (most recent call last):
        File "/tmp/user/21163/pip-build-env-4trpsjo5/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 155, in run
          self._create_wheel_file(bdist_wheel)
        File "/tmp/user/21163/pip-build-env-4trpsjo5/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 357, in _create_wheel_file
          files, mapping = self._run_build_commands(dist_name, unpacked, lib, tmp)
        File "/tmp/user/21163/pip-build-env-4trpsjo5/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 280, in _run_build_commands
          self._run_build_subcommands()
        File "/tmp/user/21163/pip-build-env-4trpsjo5/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 307, in _run_build_subcommands
          self.run_command(name)
        File "/tmp/user/21163/pip-build-env-4trpsjo5/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/tmp/user/21163/pip-build-env-4trpsjo5/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 967, in run_command
          super().run_command(command)
        File "/tmp/user/21163/pip-build-env-4trpsjo5/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/user/21163/pip-build-env-4trpsjo5/overlay/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 91, in run
          _build_ext.run(self)
        File "/tmp/user/21163/pip-build-env-4trpsjo5/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
          self.build_extensions()
        File "<string>", line 164, in build_extensions
        File "<string>", line 147, in configure
        File "/lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/lib/python3.10/subprocess.py", line 369, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', '/lfs/ampere8/0/rschaef/KoyejoLab-Revisiting-Model-Collapse/submodules/vllm', '-G', 'Ninja', '-DCMAKE_BUILD_TYPE=RelWithDebInfo', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/tmp/user/21163/tmppm34pm7i.build-lib/vllm', '-DCMAKE_ARCHIVE_OUTPUT_DIRECTORY=/tmp/user/21163/tmpt2b60pm3.build-temp', '-DVLLM_PYTHON_EXECUTABLE=/lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/bin/python3.10', '-DNVCC_THREADS=8', '-DCMAKE_JOB_POOL_COMPILE:STRING=compile', '-DCMAKE_JOB_POOLS:STRING=compile=64']' returned non-zero exit status 1.
      /tmp/user/21163/pip-build-env-4trpsjo5/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py:988: _DebuggingTips: Problem in editable installation.
      !!
RylanSchaeffer commented 7 months ago

I'm not sure why it says 12.4. I'm uninstalling and trying conda install nvidia/label/cuda-12.2.0::cuda instead.

$ nvcc --version                      
nvcc: NVIDIA (R) Cuda compiler driver                                                                                                                
Copyright (c) 2005-2023 NVIDIA Corporation                                                                                                           
Built on Tue_Jun_13_19:16:58_PDT_2023                                                                                                                
Cuda compilation tools, release 12.2, V12.2.91                                                                                                       
Build cuda_12.2.r12.2/compiler.32965470_0

This matches nvidia-smi:

| NVIDIA-SMI 535.54.03              Driver Version: 535.54.03    CUDA Version: 12.2     |

Now trying pip install -e . again.

RylanSchaeffer commented 7 months ago

Failed again:

Building wheels for collected packages: vllm
  Building editable for vllm (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building editable for vllm (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [175 lines of output]
      /tmp/user/21163/pip-build-env-rnwillx9/overlay/lib/python3.10/site-packages/torch/nn/modules/transformer.py:20: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at ../torch/csrc/utils/tensor_numpy.cpp:84.)
        device: torch.device = torch.device(torch._C._get_default_device()),  # torch.device('cpu'),
      running editable_wheel
      creating /tmp/user/21163/pip-wheel-89cl6la1/.tmp-r3mm9nv7/vllm.egg-info
      writing /tmp/user/21163/pip-wheel-89cl6la1/.tmp-r3mm9nv7/vllm.egg-info/PKG-INFO
      writing dependency_links to /tmp/user/21163/pip-wheel-89cl6la1/.tmp-r3mm9nv7/vllm.egg-info/dependency_links.txt
      writing requirements to /tmp/user/21163/pip-wheel-89cl6la1/.tmp-r3mm9nv7/vllm.egg-info/requires.txt
      writing top-level names to /tmp/user/21163/pip-wheel-89cl6la1/.tmp-r3mm9nv7/vllm.egg-info/top_level.txt
      writing manifest file '/tmp/user/21163/pip-wheel-89cl6la1/.tmp-r3mm9nv7/vllm.egg-info/SOURCES.txt'
      reading manifest file '/tmp/user/21163/pip-wheel-89cl6la1/.tmp-r3mm9nv7/vllm.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      adding license file 'LICENSE'
      writing manifest file '/tmp/user/21163/pip-wheel-89cl6la1/.tmp-r3mm9nv7/vllm.egg-info/SOURCES.txt'
      creating '/tmp/user/21163/pip-wheel-89cl6la1/.tmp-r3mm9nv7/vllm-0.3.3+cu122.dist-info'
      creating /tmp/user/21163/pip-wheel-89cl6la1/.tmp-r3mm9nv7/vllm-0.3.3+cu122.dist-info/WHEEL
      running build_py
      running build_ext
      -- The CXX compiler identification is GNU 9.4.0
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Build type: RelWithDebInfo
      -- Found Python: /lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/bin/python3.10 (found version "3.10.14") found components: Interpreter Development.Module
      -- Found python matching: /lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/bin/python3.10.
      -- Found CUDA: /lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env (found version "12.2")
      -- The CUDA compiler identification is NVIDIA 12.2.91
      -- Detecting CUDA compiler ABI info
      -- Detecting CUDA compiler ABI info - done
      -- Check for working CUDA compiler: /lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/bin/nvcc - skipped
      -- Detecting CUDA compile features
      -- Detecting CUDA compile features - done
      -- Found CUDAToolkit: /lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/include (found version "12.2.91")
      -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
      -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
      -- Looking for pthread_create in pthreads
      -- Looking for pthread_create in pthreads - not found
      -- Looking for pthread_create in pthread
      -- Looking for pthread_create in pthread - found
      -- Found Threads: TRUE
      -- Caffe2: CUDA detected: 12.2
      -- Caffe2: CUDA nvcc is: /lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/bin/nvcc
      -- Caffe2: CUDA toolkit directory: /lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env
      -- Caffe2: Header version is: 12.2
      -- /lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/lib/libnvrtc.so shorthash is 31d24739
      -- USE_CUDNN is set to 0. Compiling without cuDNN support
      -- USE_CUSPARSELT is set to 0. Compiling without cuSPARSELt support
      -- Autodetected CUDA architecture(s):  8.0 8.0 8.0 8.0 8.0 8.0 8.0 8.0
      -- Added CUDA NVCC flags for: -gencode;arch=compute_80,code=sm_80
      CMake Warning at /tmp/user/21163/pip-build-env-rnwillx9/overlay/lib/python3.10/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:22 (message):
        static library kineto_LIBRARY-NOTFOUND not found.
      Call Stack (most recent call first):
        /tmp/user/21163/pip-build-env-rnwillx9/overlay/lib/python3.10/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:127 (append_torchlib_if_found)
        CMakeLists.txt:64 (find_package)

      -- Found Torch: /tmp/user/21163/pip-build-env-rnwillx9/overlay/lib/python3.10/site-packages/torch/lib/libtorch.so
      -- CUDA supported arches: 7.0;7.5;8.0;8.6;8.9;9.0
      -- CUDA target arches: 80
      -- Punica target arches: 80
      -- Enabling C extension.
      -- Enabling moe extension.
      -- Configuring done (6.7s)
      CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
      Please set them or make sure they are set and tested correctly in the CMake files:
      LIBNVTOOLSEXT
          linked by target "_punica_C" in directory /lfs/ampere8/0/rschaef/KoyejoLab-Revisiting-Model-Collapse/submodules/vllm
          linked by target "_moe_C" in directory /lfs/ampere8/0/rschaef/KoyejoLab-Revisiting-Model-Collapse/submodules/vllm
          linked by target "_C" in directory /lfs/ampere8/0/rschaef/KoyejoLab-Revisiting-Model-Collapse/submodules/vllm

      -- Generating done (0.0s)
      CMake Generate step failed.  Build files cannot be regenerated correctly.
      Traceback (most recent call last):
        File "/tmp/user/21163/pip-build-env-rnwillx9/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 155, in run
          self._create_wheel_file(bdist_wheel)
        File "/tmp/user/21163/pip-build-env-rnwillx9/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 357, in _create_wheel_file
          files, mapping = self._run_build_commands(dist_name, unpacked, lib, tmp)
        File "/tmp/user/21163/pip-build-env-rnwillx9/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 280, in _run_build_commands
          self._run_build_subcommands()
        File "/tmp/user/21163/pip-build-env-rnwillx9/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 307, in _run_build_subcommands
          self.run_command(name)
        File "/tmp/user/21163/pip-build-env-rnwillx9/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/tmp/user/21163/pip-build-env-rnwillx9/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 967, in run_command
          super().run_command(command)
        File "/tmp/user/21163/pip-build-env-rnwillx9/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/user/21163/pip-build-env-rnwillx9/overlay/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 91, in run
          _build_ext.run(self)
        File "/tmp/user/21163/pip-build-env-rnwillx9/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
          self.build_extensions()
        File "<string>", line 164, in build_extensions
        File "<string>", line 147, in configure
        File "/lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/lib/python3.10/subprocess.py", line 369, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', '/lfs/ampere8/0/rschaef/KoyejoLab-Revisiting-Model-Collapse/submodules/vllm', '-G', 'Ninja', '-DCMAKE_BUILD_TYPE=RelWithDebInfo', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/tmp/user/21163/tmpi8zwfa0x.build-lib/vllm', '-DCMAKE_ARCHIVE_OUTPUT_DIRECTORY=/tmp/user/21163/tmptu5etr5v.build-temp', '-DVLLM_PYTHON_EXECUTABLE=/lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/bin/python3.10', '-DNVCC_THREADS=8', '-DCMAKE_JOB_POOL_COMPILE:STRING=compile', '-DCMAKE_JOB_POOLS:STRING=compile=64']' returned non-zero exit status 1.
      /tmp/user/21163/pip-build-env-rnwillx9/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py:988: _DebuggingTips: Problem in editable installation.
      !!

              ********************************************************************************
              An error happened while installing `vllm` in editable mode.

              The following steps are recommended to help debug this problem:

              - Try to install the project normally, without using the editable mode.
                Does the error still persist?
                (If it does, try fixing the problem before attempting the editable mode).
              - If you are using binary extensions, make sure you have all OS-level
                dependencies installed (e.g. compilers, toolchains, binary libraries, ...).
              - Try the latest version of setuptools (maybe the error was already fixed).
              - If you (or your project dependencies) are using any setuptools extension
                or customization, make sure they support the editable mode.

              After following the steps above, if the problem still persists and
              you think this is related to how setuptools handles editable installations,
              please submit a reproducible example
              (see https://stackoverflow.com/help/minimal-reproducible-example) to:

                  https://github.com/pypa/setuptools/issues

              See https://setuptools.pypa.io/en/latest/userguide/development_mode.html for details.
              ********************************************************************************

      !!
        cmd_obj.run()
youkaichao commented 7 months ago

CMake Warning at /tmp/user/21163/pip-build-env-rnwillx9/overlay/lib/python3.10/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:22 (message): static library kineto_LIBRARY-NOTFOUND not found.

It seems to be a problem of yout pytorch environment. How did you install pytorch?

RylanSchaeffer commented 7 months ago

How did you install pytorch?

Following the instructions on the pytorch website:

conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia

I promise I'm not trying to do something weird. I'm literally trying to install the most vanilla versions of everything.

youkaichao commented 7 months ago

You can try to use our docker image and see if it works for you: docker pull vllm/vllm-openai:v0.3.3

The script to build the image is also available https://github.com/vllm-project/vllm/blob/main/Dockerfile .

RylanSchaeffer commented 7 months ago

Here's what I just tried:

  1. I deleted my environment with conda remove -n model_collapse_env --all
  2. I purged all my conda files conda clean --all
  3. I deleted .cache: rm -rf .cache
  4. Create a new conda environment conda create -n model_collapse_env python=3.11
  5. Activate the environment: conda activate model_collapse_env
  6. Install Pytorch: conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia
  7. Check nvidia-smi - looks fine
  8. cd into the vllm directory, git pull and install again: pip install --no-cache-dir --force-reinstall -e .

The error:

$ pip install --no-cache-dir --force-reinstall -e .
Obtaining file:///lfs/ampere8/0/rschaef/KoyejoLab-Revisiting-Model-Collapse/submodules/vllm
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Collecting cmake>=3.21 (from vllm==0.3.3+cu122)
  Downloading cmake-3.28.4-py2.py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (6.3 kB)
Collecting ninja (from vllm==0.3.3+cu122)
  Downloading ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl.metadata (5.3 kB)
Collecting psutil (from vllm==0.3.3+cu122)
  Downloading psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (21 kB)
Collecting ray>=2.9 (from vllm==0.3.3+cu122)
  Downloading ray-2.10.0-cp311-cp311-manylinux2014_x86_64.whl.metadata (13 kB)
Collecting sentencepiece (from vllm==0.3.3+cu122)
  Downloading sentencepiece-0.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.7 kB)
Collecting numpy (from vllm==0.3.3+cu122)
  Downloading numpy-1.26.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (61 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.0/61.0 kB 9.7 MB/s eta 0:00:00
Collecting torch==2.1.2 (from vllm==0.3.3+cu122)
  Downloading torch-2.1.2-cp311-cp311-manylinux1_x86_64.whl.metadata (25 kB)
Collecting transformers>=4.39.1 (from vllm==0.3.3+cu122)
  Downloading transformers-4.39.1-py3-none-any.whl.metadata (134 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 134.8/134.8 kB 20.5 MB/s eta 0:00:00
Collecting xformers==0.0.23.post1 (from vllm==0.3.3+cu122)
  Downloading xformers-0.0.23.post1-cp311-cp311-manylinux2014_x86_64.whl.metadata (1.0 kB)
Collecting fastapi (from vllm==0.3.3+cu122)
  Downloading fastapi-0.110.0-py3-none-any.whl.metadata (25 kB)
Collecting uvicorn[standard] (from vllm==0.3.3+cu122)
  Downloading uvicorn-0.29.0-py3-none-any.whl.metadata (6.3 kB)
Collecting pydantic>=2.0 (from vllm==0.3.3+cu122)
  Downloading pydantic-2.6.4-py3-none-any.whl.metadata (85 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 85.1/85.1 kB 130.5 MB/s eta 0:00:00
Collecting prometheus-client>=0.18.0 (from vllm==0.3.3+cu122)
  Downloading prometheus_client-0.20.0-py3-none-any.whl.metadata (1.8 kB)
Collecting pynvml==11.5.0 (from vllm==0.3.3+cu122)
  Downloading pynvml-11.5.0-py3-none-any.whl.metadata (7.8 kB)
Collecting triton>=2.1.0 (from vllm==0.3.3+cu122)
  Downloading triton-2.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.4 kB)
Collecting outlines==0.0.34 (from vllm==0.3.3+cu122)
  Downloading outlines-0.0.34-py3-none-any.whl.metadata (13 kB)
Collecting cupy-cuda12x==12.1.0 (from vllm==0.3.3+cu122)
  Downloading cupy_cuda12x-12.1.0-cp311-cp311-manylinux2014_x86_64.whl.metadata (2.6 kB)
Collecting fastrlock>=0.5 (from cupy-cuda12x==12.1.0->vllm==0.3.3+cu122)
  Downloading fastrlock-0.8.2-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_28_x86_64.whl.metadata (9.3 kB)
Collecting interegular (from outlines==0.0.34->vllm==0.3.3+cu122)
  Downloading interegular-0.3.3-py37-none-any.whl.metadata (3.0 kB)
Collecting jinja2 (from outlines==0.0.34->vllm==0.3.3+cu122)
  Downloading Jinja2-3.1.3-py3-none-any.whl.metadata (3.3 kB)
Collecting lark (from outlines==0.0.34->vllm==0.3.3+cu122)
  Downloading lark-1.1.9-py3-none-any.whl.metadata (1.9 kB)
Collecting nest-asyncio (from outlines==0.0.34->vllm==0.3.3+cu122)
  Downloading nest_asyncio-1.6.0-py3-none-any.whl.metadata (2.8 kB)
Collecting cloudpickle (from outlines==0.0.34->vllm==0.3.3+cu122)
  Downloading cloudpickle-3.0.0-py3-none-any.whl.metadata (7.0 kB)
Collecting diskcache (from outlines==0.0.34->vllm==0.3.3+cu122)
  Downloading diskcache-5.6.3-py3-none-any.whl.metadata (20 kB)
Collecting scipy (from outlines==0.0.34->vllm==0.3.3+cu122)
  Downloading scipy-1.12.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (60 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.4/60.4 kB 154.2 MB/s eta 0:00:00
Collecting numba (from outlines==0.0.34->vllm==0.3.3+cu122)
  Downloading numba-0.59.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (2.7 kB)
Collecting joblib (from outlines==0.0.34->vllm==0.3.3+cu122)
  Downloading joblib-1.3.2-py3-none-any.whl.metadata (5.4 kB)
Collecting referencing (from outlines==0.0.34->vllm==0.3.3+cu122)
  Downloading referencing-0.34.0-py3-none-any.whl.metadata (2.8 kB)
Collecting jsonschema (from outlines==0.0.34->vllm==0.3.3+cu122)
  Downloading jsonschema-4.21.1-py3-none-any.whl.metadata (7.8 kB)
Collecting requests (from outlines==0.0.34->vllm==0.3.3+cu122)
  Downloading requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)
Collecting filelock (from torch==2.1.2->vllm==0.3.3+cu122)
  Downloading filelock-3.13.1-py3-none-any.whl.metadata (2.8 kB)
Collecting typing-extensions (from torch==2.1.2->vllm==0.3.3+cu122)
  Downloading typing_extensions-4.10.0-py3-none-any.whl.metadata (3.0 kB)
Collecting sympy (from torch==2.1.2->vllm==0.3.3+cu122)
  Downloading sympy-1.12-py3-none-any.whl.metadata (12 kB)
Collecting networkx (from torch==2.1.2->vllm==0.3.3+cu122)
  Downloading networkx-3.2.1-py3-none-any.whl.metadata (5.2 kB)
Collecting fsspec (from torch==2.1.2->vllm==0.3.3+cu122)
  Downloading fsspec-2024.3.1-py3-none-any.whl.metadata (6.8 kB)
Collecting nvidia-cuda-nvrtc-cu12==12.1.105 (from torch==2.1.2->vllm==0.3.3+cu122)
  Downloading nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cuda-runtime-cu12==12.1.105 (from torch==2.1.2->vllm==0.3.3+cu122)
  Downloading nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cuda-cupti-cu12==12.1.105 (from torch==2.1.2->vllm==0.3.3+cu122)
  Downloading nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-cudnn-cu12==8.9.2.26 (from torch==2.1.2->vllm==0.3.3+cu122)
  Downloading nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-cublas-cu12==12.1.3.1 (from torch==2.1.2->vllm==0.3.3+cu122)
  Downloading nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cufft-cu12==11.0.2.54 (from torch==2.1.2->vllm==0.3.3+cu122)
  Downloading nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-curand-cu12==10.3.2.106 (from torch==2.1.2->vllm==0.3.3+cu122)
  Downloading nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cusolver-cu12==11.4.5.107 (from torch==2.1.2->vllm==0.3.3+cu122)
  Downloading nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-cusparse-cu12==12.1.0.106 (from torch==2.1.2->vllm==0.3.3+cu122)
  Downloading nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-nccl-cu12==2.18.1 (from torch==2.1.2->vllm==0.3.3+cu122)
  Downloading nvidia_nccl_cu12-2.18.1-py3-none-manylinux1_x86_64.whl.metadata (1.8 kB)
Collecting nvidia-nvtx-cu12==12.1.105 (from torch==2.1.2->vllm==0.3.3+cu122)
  Downloading nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.7 kB)
Collecting triton>=2.1.0 (from vllm==0.3.3+cu122)
  Downloading triton-2.1.0-0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.3 kB)
Collecting nvidia-nvjitlink-cu12 (from nvidia-cusolver-cu12==11.4.5.107->torch==2.1.2->vllm==0.3.3+cu122)
  Downloading nvidia_nvjitlink_cu12-12.4.99-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)
Collecting annotated-types>=0.4.0 (from pydantic>=2.0->vllm==0.3.3+cu122)
  Downloading annotated_types-0.6.0-py3-none-any.whl.metadata (12 kB)
Collecting pydantic-core==2.16.3 (from pydantic>=2.0->vllm==0.3.3+cu122)
  Downloading pydantic_core-2.16.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.5 kB)
Collecting click>=7.0 (from ray>=2.9->vllm==0.3.3+cu122)
  Downloading click-8.1.7-py3-none-any.whl.metadata (3.0 kB)
Collecting msgpack<2.0.0,>=1.0.0 (from ray>=2.9->vllm==0.3.3+cu122)
  Downloading msgpack-1.0.8-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (9.1 kB)
Collecting packaging (from ray>=2.9->vllm==0.3.3+cu122)
  Downloading packaging-24.0-py3-none-any.whl.metadata (3.2 kB)
Collecting protobuf!=3.19.5,>=3.15.3 (from ray>=2.9->vllm==0.3.3+cu122)
  Downloading protobuf-5.26.0-cp37-abi3-manylinux2014_x86_64.whl.metadata (592 bytes)
Collecting pyyaml (from ray>=2.9->vllm==0.3.3+cu122)
  Downloading PyYAML-6.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.1 kB)
Collecting aiosignal (from ray>=2.9->vllm==0.3.3+cu122)
  Downloading aiosignal-1.3.1-py3-none-any.whl.metadata (4.0 kB)
Collecting frozenlist (from ray>=2.9->vllm==0.3.3+cu122)
  Downloading frozenlist-1.4.1-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB)
Collecting huggingface-hub<1.0,>=0.19.3 (from transformers>=4.39.1->vllm==0.3.3+cu122)
  Downloading huggingface_hub-0.21.4-py3-none-any.whl.metadata (13 kB)
Collecting regex!=2019.12.17 (from transformers>=4.39.1->vllm==0.3.3+cu122)
  Downloading regex-2023.12.25-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (40 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.9/40.9 kB 113.1 MB/s eta 0:00:00
Collecting tokenizers<0.19,>=0.14 (from transformers>=4.39.1->vllm==0.3.3+cu122)
  Downloading tokenizers-0.15.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB)
Collecting safetensors>=0.4.1 (from transformers>=4.39.1->vllm==0.3.3+cu122)
  Downloading safetensors-0.4.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.8 kB)
Collecting tqdm>=4.27 (from transformers>=4.39.1->vllm==0.3.3+cu122)
  Downloading tqdm-4.66.2-py3-none-any.whl.metadata (57 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.6/57.6 kB 131.0 MB/s eta 0:00:00
Collecting starlette<0.37.0,>=0.36.3 (from fastapi->vllm==0.3.3+cu122)
  Downloading starlette-0.36.3-py3-none-any.whl.metadata (5.9 kB)
Collecting h11>=0.8 (from uvicorn[standard]->vllm==0.3.3+cu122)
  Downloading h11-0.14.0-py3-none-any.whl.metadata (8.2 kB)
Collecting httptools>=0.5.0 (from uvicorn[standard]->vllm==0.3.3+cu122)
  Downloading httptools-0.6.1-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.6 kB)
Collecting python-dotenv>=0.13 (from uvicorn[standard]->vllm==0.3.3+cu122)
  Downloading python_dotenv-1.0.1-py3-none-any.whl.metadata (23 kB)
Collecting uvloop!=0.15.0,!=0.15.1,>=0.14.0 (from uvicorn[standard]->vllm==0.3.3+cu122)
  Downloading uvloop-0.19.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.9 kB)
Collecting watchfiles>=0.13 (from uvicorn[standard]->vllm==0.3.3+cu122)
  Downloading watchfiles-0.21.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.9 kB)
Collecting websockets>=10.4 (from uvicorn[standard]->vllm==0.3.3+cu122)
  Downloading websockets-12.0-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)
Collecting anyio<5,>=3.4.0 (from starlette<0.37.0,>=0.36.3->fastapi->vllm==0.3.3+cu122)
  Downloading anyio-4.3.0-py3-none-any.whl.metadata (4.6 kB)
Collecting MarkupSafe>=2.0 (from jinja2->outlines==0.0.34->vllm==0.3.3+cu122)
  Downloading MarkupSafe-2.1.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.0 kB)
Collecting attrs>=22.2.0 (from jsonschema->outlines==0.0.34->vllm==0.3.3+cu122)
  Downloading attrs-23.2.0-py3-none-any.whl.metadata (9.5 kB)
Collecting jsonschema-specifications>=2023.03.6 (from jsonschema->outlines==0.0.34->vllm==0.3.3+cu122)
  Downloading jsonschema_specifications-2023.12.1-py3-none-any.whl.metadata (3.0 kB)
Collecting rpds-py>=0.7.1 (from jsonschema->outlines==0.0.34->vllm==0.3.3+cu122)
  Downloading rpds_py-0.18.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.1 kB)
Collecting llvmlite<0.43,>=0.42.0dev0 (from numba->outlines==0.0.34->vllm==0.3.3+cu122)
  Downloading llvmlite-0.42.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.8 kB)
Collecting charset-normalizer<4,>=2 (from requests->outlines==0.0.34->vllm==0.3.3+cu122)
  Downloading charset_normalizer-3.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (33 kB)
Collecting idna<4,>=2.5 (from requests->outlines==0.0.34->vllm==0.3.3+cu122)
  Downloading idna-3.6-py3-none-any.whl.metadata (9.9 kB)
Collecting urllib3<3,>=1.21.1 (from requests->outlines==0.0.34->vllm==0.3.3+cu122)
  Downloading urllib3-2.2.1-py3-none-any.whl.metadata (6.4 kB)
Collecting certifi>=2017.4.17 (from requests->outlines==0.0.34->vllm==0.3.3+cu122)
  Downloading certifi-2024.2.2-py3-none-any.whl.metadata (2.2 kB)
Collecting mpmath>=0.19 (from sympy->torch==2.1.2->vllm==0.3.3+cu122)
  Downloading mpmath-1.3.0-py3-none-any.whl.metadata (8.6 kB)
Collecting sniffio>=1.1 (from anyio<5,>=3.4.0->starlette<0.37.0,>=0.36.3->fastapi->vllm==0.3.3+cu122)
  Downloading sniffio-1.3.1-py3-none-any.whl.metadata (3.9 kB)
Downloading cupy_cuda12x-12.1.0-cp311-cp311-manylinux2014_x86_64.whl (83.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 83.5/83.5 MB 155.4 MB/s eta 0:00:00
Downloading outlines-0.0.34-py3-none-any.whl (76 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 76.5/76.5 kB 145.6 MB/s eta 0:00:00
Downloading pynvml-11.5.0-py3-none-any.whl (53 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.1/53.1 kB 129.1 MB/s eta 0:00:00
Downloading torch-2.1.2-cp311-cp311-manylinux1_x86_64.whl (670.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 670.2/670.2 MB 203.2 MB/s eta 0:00:00
Downloading triton-2.1.0-0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (89.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 89.2/89.2 MB 200.1 MB/s eta 0:00:00
Downloading xformers-0.0.23.post1-cp311-cp311-manylinux2014_x86_64.whl (213.0 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 213.0/213.0 MB 166.5 MB/s eta 0:00:00
Downloading nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl (410.6 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 410.6/410.6 MB 223.2 MB/s eta 0:00:00
Downloading nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (14.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.1/14.1 MB 226.2 MB/s eta 0:00:00
Downloading nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (23.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23.7/23.7 MB 211.4 MB/s eta 0:00:00
Downloading nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (823 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 823.6/823.6 kB 175.8 MB/s eta 0:00:00
Downloading nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl (731.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 731.7/731.7 MB 228.1 MB/s eta 0:00:00
Downloading nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl (121.6 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 121.6/121.6 MB 224.1 MB/s eta 0:00:00
Downloading nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl (56.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56.5/56.5 MB 162.2 MB/s eta 0:00:00
Downloading nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl (124.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 124.2/124.2 MB 195.3 MB/s eta 0:00:00
Downloading nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl (196.0 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 196.0/196.0 MB 149.3 MB/s eta 0:00:00
Downloading nvidia_nccl_cu12-2.18.1-py3-none-manylinux1_x86_64.whl (209.8 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 209.8/209.8 MB 163.6 MB/s eta 0:00:00
Downloading nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (99 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 99.1/99.1 kB 151.0 MB/s eta 0:00:00
Downloading cmake-3.28.4-py2.py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (26.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 26.3/26.3 MB 164.6 MB/s eta 0:00:00
Downloading numpy-1.26.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18.3/18.3 MB 166.4 MB/s eta 0:00:00
Downloading prometheus_client-0.20.0-py3-none-any.whl (54 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.5/54.5 kB 176.4 MB/s eta 0:00:00
Downloading pydantic-2.6.4-py3-none-any.whl (394 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 394.9/394.9 kB 223.6 MB/s eta 0:00:00
Downloading pydantic_core-2.16.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.2/2.2 MB 179.2 MB/s eta 0:00:00
Downloading ray-2.10.0-cp311-cp311-manylinux2014_x86_64.whl (65.6 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 65.6/65.6 MB 168.1 MB/s eta 0:00:00
Downloading transformers-4.39.1-py3-none-any.whl (8.8 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.8/8.8 MB 175.7 MB/s eta 0:00:00
Downloading fastapi-0.110.0-py3-none-any.whl (92 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 92.1/92.1 kB 160.7 MB/s eta 0:00:00
Downloading ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl (307 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 307.2/307.2 kB 170.5 MB/s eta 0:00:00
Downloading psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (288 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 288.2/288.2 kB 162.2 MB/s eta 0:00:00
Downloading sentencepiece-0.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 202.6 MB/s eta 0:00:00
Downloading annotated_types-0.6.0-py3-none-any.whl (12 kB)
Downloading click-8.1.7-py3-none-any.whl (97 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 97.9/97.9 kB 147.6 MB/s eta 0:00:00
Downloading fastrlock-0.8.2-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_28_x86_64.whl (52 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 52.6/52.6 kB 140.4 MB/s eta 0:00:00
Downloading h11-0.14.0-py3-none-any.whl (58 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.3/58.3 kB 147.4 MB/s eta 0:00:00
Downloading httptools-0.6.1-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (318 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 318.5/318.5 kB 169.8 MB/s eta 0:00:00
Downloading huggingface_hub-0.21.4-py3-none-any.whl (346 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 346.4/346.4 kB 212.2 MB/s eta 0:00:00
Downloading fsspec-2024.3.1-py3-none-any.whl (171 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 172.0/172.0 kB 215.5 MB/s eta 0:00:00
Downloading msgpack-1.0.8-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (409 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 409.3/409.3 kB 175.6 MB/s eta 0:00:00
Downloading packaging-24.0-py3-none-any.whl (53 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.5/53.5 kB 139.6 MB/s eta 0:00:00
Downloading protobuf-5.26.0-cp37-abi3-manylinux2014_x86_64.whl (302 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 302.8/302.8 kB 172.7 MB/s eta 0:00:00
Downloading python_dotenv-1.0.1-py3-none-any.whl (19 kB)
Downloading PyYAML-6.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (757 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 757.7/757.7 kB 212.8 MB/s eta 0:00:00
Downloading regex-2023.12.25-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (785 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 785.1/785.1 kB 213.4 MB/s eta 0:00:00
Downloading safetensors-0.4.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 213.1 MB/s eta 0:00:00
Downloading starlette-0.36.3-py3-none-any.whl (71 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 71.5/71.5 kB 146.4 MB/s eta 0:00:00
Downloading tokenizers-0.15.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.6 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.6/3.6 MB 179.9 MB/s eta 0:00:00
Downloading tqdm-4.66.2-py3-none-any.whl (78 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.3/78.3 kB 150.6 MB/s eta 0:00:00
Downloading typing_extensions-4.10.0-py3-none-any.whl (33 kB)
Downloading uvloop-0.19.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.5/3.5 MB 180.8 MB/s eta 0:00:00
Downloading watchfiles-0.21.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 174.5 MB/s eta 0:00:00
Downloading websockets-12.0-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (130 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 130.9/130.9 kB 162.0 MB/s eta 0:00:00
Downloading aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
Downloading frozenlist-1.4.1-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (272 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 272.3/272.3 kB 161.1 MB/s eta 0:00:00
Downloading cloudpickle-3.0.0-py3-none-any.whl (20 kB)
Downloading diskcache-5.6.3-py3-none-any.whl (45 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45.5/45.5 kB 168.0 MB/s eta 0:00:00
Downloading filelock-3.13.1-py3-none-any.whl (11 kB)
Downloading interegular-0.3.3-py37-none-any.whl (23 kB)
Downloading Jinja2-3.1.3-py3-none-any.whl (133 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.2/133.2 kB 169.2 MB/s eta 0:00:00
Downloading joblib-1.3.2-py3-none-any.whl (302 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 302.2/302.2 kB 164.8 MB/s eta 0:00:00
Downloading jsonschema-4.21.1-py3-none-any.whl (85 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 85.5/85.5 kB 189.3 MB/s eta 0:00:00
Downloading referencing-0.34.0-py3-none-any.whl (26 kB)
Downloading lark-1.1.9-py3-none-any.whl (111 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 111.7/111.7 kB 152.4 MB/s eta 0:00:00
Downloading nest_asyncio-1.6.0-py3-none-any.whl (5.2 kB)
Downloading networkx-3.2.1-py3-none-any.whl (1.6 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 170.3 MB/s eta 0:00:00
Downloading numba-0.59.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.7/3.7 MB 175.3 MB/s eta 0:00:00
Downloading requests-2.31.0-py3-none-any.whl (62 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.6/62.6 kB 183.6 MB/s eta 0:00:00
Downloading scipy-1.12.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (38.4 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 38.4/38.4 MB 169.1 MB/s eta 0:00:00
Downloading sympy-1.12-py3-none-any.whl (5.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.7/5.7 MB 178.8 MB/s eta 0:00:00
Downloading uvicorn-0.29.0-py3-none-any.whl (60 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.8/60.8 kB 146.7 MB/s eta 0:00:00
Downloading anyio-4.3.0-py3-none-any.whl (85 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 85.6/85.6 kB 154.5 MB/s eta 0:00:00
Downloading attrs-23.2.0-py3-none-any.whl (60 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.8/60.8 kB 147.5 MB/s eta 0:00:00
Downloading certifi-2024.2.2-py3-none-any.whl (163 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 163.8/163.8 kB 215.2 MB/s eta 0:00:00
Downloading charset_normalizer-3.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (140 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 140.3/140.3 kB 199.7 MB/s eta 0:00:00
Downloading idna-3.6-py3-none-any.whl (61 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.6/61.6 kB 181.5 MB/s eta 0:00:00
Downloading jsonschema_specifications-2023.12.1-py3-none-any.whl (18 kB)
Downloading llvmlite-0.42.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (43.8 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 43.8/43.8 MB 169.1 MB/s eta 0:00:00
Downloading MarkupSafe-2.1.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (28 kB)
Downloading mpmath-1.3.0-py3-none-any.whl (536 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 536.2/536.2 kB 221.8 MB/s eta 0:00:00
Downloading rpds_py-0.18.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 164.9 MB/s eta 0:00:00
Downloading urllib3-2.2.1-py3-none-any.whl (121 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 121.1/121.1 kB 158.8 MB/s eta 0:00:00
Downloading nvidia_nvjitlink_cu12-12.4.99-py3-none-manylinux2014_x86_64.whl (21.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 21.1/21.1 MB 170.4 MB/s eta 0:00:00
Downloading sniffio-1.3.1-py3-none-any.whl (10 kB)
Building wheels for collected packages: vllm
  Building editable for vllm (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building editable for vllm (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [152 lines of output]
      /tmp/user/21163/pip-build-env-vvupu6fr/overlay/lib/python3.11/site-packages/torch/nn/modules/transformer.py:20: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at ../torch/csrc/utils/tensor_numpy.cpp:84.)
        device: torch.device = torch.device(torch._C._get_default_device()),  # torch.device('cpu'),
      running editable_wheel
      creating /tmp/user/21163/pip-wheel-wim8vklg/.tmp-bp23gjik/vllm.egg-info
      writing /tmp/user/21163/pip-wheel-wim8vklg/.tmp-bp23gjik/vllm.egg-info/PKG-INFO
      writing dependency_links to /tmp/user/21163/pip-wheel-wim8vklg/.tmp-bp23gjik/vllm.egg-info/dependency_links.txt
      writing requirements to /tmp/user/21163/pip-wheel-wim8vklg/.tmp-bp23gjik/vllm.egg-info/requires.txt
      writing top-level names to /tmp/user/21163/pip-wheel-wim8vklg/.tmp-bp23gjik/vllm.egg-info/top_level.txt
      writing manifest file '/tmp/user/21163/pip-wheel-wim8vklg/.tmp-bp23gjik/vllm.egg-info/SOURCES.txt'
      reading manifest file '/tmp/user/21163/pip-wheel-wim8vklg/.tmp-bp23gjik/vllm.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      adding license file 'LICENSE'
      writing manifest file '/tmp/user/21163/pip-wheel-wim8vklg/.tmp-bp23gjik/vllm.egg-info/SOURCES.txt'
      creating '/tmp/user/21163/pip-wheel-wim8vklg/.tmp-bp23gjik/vllm-0.3.3+cu122.dist-info'
      creating /tmp/user/21163/pip-wheel-wim8vklg/.tmp-bp23gjik/vllm-0.3.3+cu122.dist-info/WHEEL
      running build_py
      running build_ext
      -- The CXX compiler identification is GNU 9.4.0
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Build type: RelWithDebInfo
      -- Found Python: /lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/bin/python (found version "3.11.8") found components: Interpreter Development.Module
      -- Found python matching: /lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/bin/python.
      -- Found CUDA: /usr/local/cuda (found version "12.2")
      -- The CUDA compiler identification is unknown
      CMake Error at /tmp/user/21163/pip-build-env-vvupu6fr/overlay/lib/python3.11/site-packages/cmake/data/share/cmake-3.28/Modules/CMakeDetermineCUDACompiler.cmake:270 (message):
        Failed to detect a default CUDA architecture.

        Compiler output:

      Call Stack (most recent call first):
        /tmp/user/21163/pip-build-env-vvupu6fr/overlay/lib/python3.11/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:47 (enable_language)
        /tmp/user/21163/pip-build-env-vvupu6fr/overlay/lib/python3.11/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:87 (include)
        /tmp/user/21163/pip-build-env-vvupu6fr/overlay/lib/python3.11/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
        CMakeLists.txt:64 (find_package)

      -- Configuring incomplete, errors occurred!
      Traceback (most recent call last):
        File "/tmp/user/21163/pip-build-env-vvupu6fr/overlay/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 155, in run
          self._create_wheel_file(bdist_wheel)
        File "/tmp/user/21163/pip-build-env-vvupu6fr/overlay/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 357, in _create_wheel_file
          files, mapping = self._run_build_commands(dist_name, unpacked, lib, tmp)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/user/21163/pip-build-env-vvupu6fr/overlay/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 280, in _run_build_commands
          self._run_build_subcommands()
        File "/tmp/user/21163/pip-build-env-vvupu6fr/overlay/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 307, in _run_build_subcommands
          self.run_command(name)
        File "/tmp/user/21163/pip-build-env-vvupu6fr/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/tmp/user/21163/pip-build-env-vvupu6fr/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 967, in run_command
          super().run_command(command)
        File "/tmp/user/21163/pip-build-env-vvupu6fr/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/user/21163/pip-build-env-vvupu6fr/overlay/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 91, in run
          _build_ext.run(self)
        File "/tmp/user/21163/pip-build-env-vvupu6fr/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
          self.build_extensions()
        File "<string>", line 164, in build_extensions
        File "<string>", line 147, in configure
        File "/lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/lib/python3.11/subprocess.py", line 413, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', '/lfs/ampere8/0/rschaef/KoyejoLab-Revisiting-Model-Collapse/submodules/vllm', '-G', 'Ninja', '-DCMAKE_BUILD_TYPE=RelWithDebInfo', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/tmp/user/21163/tmp00n843y3.build-lib/vllm', '-DCMAKE_ARCHIVE_OUTPUT_DIRECTORY=/tmp/user/21163/tmpov687z8o.build-temp', '-DVLLM_PYTHON_EXECUTABLE=/lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/bin/python', '-DNVCC_THREADS=8', '-DCMAKE_JOB_POOL_COMPILE:STRING=compile', '-DCMAKE_JOB_POOLS:STRING=compile=64']' returned non-zero exit status 1.
      /tmp/user/21163/pip-build-env-vvupu6fr/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py:988: _DebuggingTips: Problem in editable installation.
      !!

              ********************************************************************************
              An error happened while installing `vllm` in editable mode.

              The following steps are recommended to help debug this problem:

              - Try to install the project normally, without using the editable mode.
                Does the error still persist?
                (If it does, try fixing the problem before attempting the editable mode).
              - If you are using binary extensions, make sure you have all OS-level
                dependencies installed (e.g. compilers, toolchains, binary libraries, ...).
              - Try the latest version of setuptools (maybe the error was already fixed).
              - If you (or your project dependencies) are using any setuptools extension
                or customization, make sure they support the editable mode.

              After following the steps above, if the problem still persists and
              you think this is related to how setuptools handles editable installations,
              please submit a reproducible example
              (see https://stackoverflow.com/help/minimal-reproducible-example) to:

                  https://github.com/pypa/setuptools/issues

              See https://setuptools.pypa.io/en/latest/userguide/development_mode.html for details.
              ********************************************************************************

      !!
RylanSchaeffer commented 7 months ago

I don't understand how installing pytorch-cuda doesn't install cuda, but I am now going to try conda install nvidia/label/cuda-12.1.0::cuda and try installing vllm again. I'll report back in a few minutes

RylanSchaeffer commented 7 months ago

Error:

Building wheels for collected packages: vllm
  Building editable for vllm (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building editable for vllm (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [155 lines of output]
      /tmp/user/21163/pip-build-env-igskn0fq/overlay/lib/python3.11/site-packages/torch/nn/modules/transformer.py:20: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at ../torch/csrc/utils/tensor_numpy.cpp:84.)
        device: torch.device = torch.device(torch._C._get_default_device()),  # torch.device('cpu'),
      running editable_wheel
      creating /tmp/user/21163/pip-wheel-2svgs4gy/.tmp-454i10h6/vllm.egg-info
      writing /tmp/user/21163/pip-wheel-2svgs4gy/.tmp-454i10h6/vllm.egg-info/PKG-INFO
      writing dependency_links to /tmp/user/21163/pip-wheel-2svgs4gy/.tmp-454i10h6/vllm.egg-info/dependency_links.txt
      writing requirements to /tmp/user/21163/pip-wheel-2svgs4gy/.tmp-454i10h6/vllm.egg-info/requires.txt
      writing top-level names to /tmp/user/21163/pip-wheel-2svgs4gy/.tmp-454i10h6/vllm.egg-info/top_level.txt
      writing manifest file '/tmp/user/21163/pip-wheel-2svgs4gy/.tmp-454i10h6/vllm.egg-info/SOURCES.txt'
      reading manifest file '/tmp/user/21163/pip-wheel-2svgs4gy/.tmp-454i10h6/vllm.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      adding license file 'LICENSE'
      writing manifest file '/tmp/user/21163/pip-wheel-2svgs4gy/.tmp-454i10h6/vllm.egg-info/SOURCES.txt'
      creating '/tmp/user/21163/pip-wheel-2svgs4gy/.tmp-454i10h6/vllm-0.3.3.dist-info'
      creating /tmp/user/21163/pip-wheel-2svgs4gy/.tmp-454i10h6/vllm-0.3.3.dist-info/WHEEL
      running build_py
      running build_ext
      -- The CXX compiler identification is GNU 9.4.0
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Build type: RelWithDebInfo
      -- Found Python: /lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/bin/python (found version "3.11.8") found components: Interpreter Development.Module
      -- Found python matching: /lfs/ampere8/0/rschaef/miniconda3/envs/model_collapse_env/bin/python.
      -- Could NOT find CUDA (missing: CUDA_CUDART_LIBRARY) (found version "12.1")
      CMake Warning at /tmp/user/21163/pip-build-env-igskn0fq/overlay/lib/python3.11/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:31 (message):
        Caffe2: CUDA cannot be found.  Depending on whether you are building Caffe2
        or a Caffe2 dependent library, the next warning / error will give you more
        info.
      Call Stack (most recent call first):
        /tmp/user/21163/pip-build-env-igskn0fq/overlay/lib/python3.11/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:87 (include)
        /tmp/user/21163/pip-build-env-igskn0fq/overlay/lib/python3.11/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
        CMakeLists.txt:64 (find_package)

      CMake Error at /tmp/user/21163/pip-build-env-igskn0fq/overlay/lib/python3.11/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:91 (message):
        Your installed Caffe2 version uses CUDA but I cannot find the CUDA
        libraries.  Please set the proper CUDA prefixes and / or install CUDA.
      Call Stack (most recent call first):
        /tmp/user/21163/pip-build-env-igskn0fq/overlay/lib/python3.11/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
        CMakeLists.txt:64 (find_package)

For more info, nvcc --version exists: $ nvcc --version nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2023 NVIDIA Corporation Built on Tue_Feb__7_19:32:13_PST_2023 Cuda compilation tools, release 12.1, V12.1.66 Build cuda_12.1.r12.1/compiler.32415258_0

RylanSchaeffer commented 7 months ago

If I do conda list, I see what appears to be CUDA:

cuda                      12.1.0                        0    nvidia/label/cuda-12.1.0
cuda-cccl                 12.1.55                       0    nvidia/label/cuda-12.1.0
cuda-command-line-tools   12.1.0                        0    nvidia/label/cuda-12.1.0
cuda-compiler             12.1.0                        0    nvidia/label/cuda-12.1.0
cuda-cudart               12.1.105                      0    nvidia
cuda-cudart-dev           12.1.55                       0    nvidia/label/cuda-12.1.0
cuda-cudart-static        12.1.55                       0    nvidia/label/cuda-12.1.0
cuda-cuobjdump            12.1.55                       0    nvidia/label/cuda-12.1.0
cuda-cupti                12.1.105                      0    nvidia
cuda-cupti-static         12.1.62                       0    nvidia/label/cuda-12.1.0
cuda-cuxxfilt             12.1.55                       0    nvidia/label/cuda-12.1.0
cuda-demo-suite           12.1.55                       0    nvidia/label/cuda-12.1.0
cuda-documentation        12.1.55                       0    nvidia/label/cuda-12.1.0
cuda-driver-dev           12.1.55                       0    nvidia/label/cuda-12.1.0
cuda-gdb                  12.1.55                       0    nvidia/label/cuda-12.1.0
cuda-libraries            12.1.0                        0    nvidia
cuda-libraries-dev        12.1.0                        0    nvidia/label/cuda-12.1.0
cuda-libraries-static     12.1.0                        0    nvidia/label/cuda-12.1.0
cuda-nsight               12.1.55                       0    nvidia/label/cuda-12.1.0
cuda-nsight-compute       12.1.0                        0    nvidia/label/cuda-12.1.0
cuda-nvcc                 12.1.66                       0    nvidia/label/cuda-12.1.0
cuda-nvdisasm             12.1.55                       0    nvidia/label/cuda-12.1.0
cuda-nvml-dev             12.1.55                       0    nvidia/label/cuda-12.1.0
cuda-nvprof               12.1.55                       0    nvidia/label/cuda-12.1.0
cuda-nvprune              12.1.55                       0    nvidia/label/cuda-12.1.0
cuda-nvrtc                12.1.105                      0    nvidia
cuda-nvrtc-dev            12.1.55                       0    nvidia/label/cuda-12.1.0
cuda-nvrtc-static         12.1.55                       0    nvidia/label/cuda-12.1.0
cuda-nvtx                 12.1.105                      0    nvidia
cuda-nvvp                 12.1.55                       0    nvidia/label/cuda-12.1.0
cuda-opencl               12.4.99                       0    nvidia
cuda-opencl-dev           12.1.56                       0    nvidia/label/cuda-12.1.0
cuda-profiler-api         12.1.55                       0    nvidia/label/cuda-12.1.0
cuda-runtime              12.1.0                        0    nvidia
cuda-sanitizer-api        12.1.55                       0    nvidia/label/cuda-12.1.0
cuda-toolkit              12.1.0                        0    nvidia/label/cuda-12.1.0
cuda-tools                12.1.0                        0    nvidia/label/cuda-12.1.0
cuda-visual-tools         12.1.0                        0    nvidia/label/cuda-12.1.0
youkaichao commented 7 months ago

Your environment is complicated and might be broken. Please try the docker image.

RylanSchaeffer commented 7 months ago

I found a solution!!! I just followed https://docs.vllm.ai/en/latest/getting_started/installation.html

pip install vllm worked right out of the box!!

youkaichao commented 7 months ago

For people encountering undefined symbol problem:

TL,DR; please pip install vllm in a fresh new python environment.

The long answer: this is usually caused by an incompatible pytorch version. By default pip install vllm will install pre-built binary wheels, which is compiled against a particular pytorch version. Unfortunately, compiled binary objects are not portable across pytorch versions. Either have a fresh new environment and let pip choose the correct pytorch version for you, or you can build vllm from source.

tjingrant commented 7 months ago

I ran into the same issue (LIBNVTOOLSEXT not set) when installing from source.

I fixed it by installing using the v0.3.3 version of the repo. I think this is a problem with the latest main branch.

tanguofu commented 7 months ago

when build from source, pytorch 2.2 cuda 12.1 :

    from vllm._C import ops
ImportError: /workspace/vllm/vllm/_C.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN2at4_ops15to_dtype_layout4callERKNS_6TensorEN3c108optionalINS5_10ScalarTypeEEENS6_INS5_6LayoutEEENS6_INS5_6DeviceEEENS6_IbEEbbNS6_INS5_12MemoryFormatEEE

this is miss the ref : at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) so is there somewhere to use this ops?

youkaichao commented 7 months ago

pytorch 2.2 cuda 12.1

pt 2.2 is not supported yet.

ibicdev commented 7 months ago

I ran into the same issue (LIBNVTOOLSEXT not set) when installing from source.

I fixed it by installing using the v0.3.3 version of the repo. I think this is a problem with the latest main branch.

I had exactly the same issue (LIBNVTOOLSEXT not set when building from source) and rolling back to v0.3.3 solved it. Most likely related to the switch to cmake build (#2830).

tanguofu commented 7 months ago

pytorch 2.2 cuda 12.1

pt 2.2 is not supported yet.

when will the pytorch 2.2 will be supported ? or how to fix this error ? @youkaichao 

    from vllm._C import ops
ImportError: /workspace/vllm/vllm/_C.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN2at4_ops15to_dtype_layout4callERKNS_6TensorEN3c108optionalINS5_10ScalarTypeEEENS6_INS5_6LayoutEEENS6_INS5_6DeviceEEENS6_IbEEbbNS6_INS5_12MemoryFormatEEE
brando90 commented 7 months ago

For people encountering undefined symbol problem:

TL,DR; please pip install vllm in a fresh new python environment.

The long answer: this is usually caused by an incompatible pytorch version. By default pip install vllm will install pre-built binary wheels, which is compiled against a particular pytorch version. Unfortunately, compiled binary objects are not portable across pytorch versions. Either have a fresh new environment and let pip choose the correct pytorch version for you, or you can build vllm from source.

@youkaichao can you give us a hint of which pytorch version does work? ranges? the highest? anything?

brando90 commented 7 months ago

can we request pytorch 2.2? It's the fastest! https://github.com/vllm-project/vllm/issues/3742

brando90 commented 7 months ago

@youkaichao I followed @RylanSchaeffer advice and I still get an eror:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
outlines 0.0.37 requires pydantic>=2.0, but you have pydantic 1.10.13 which is incompatible.
Successfully installed GitPython-3.1.42 Pygments-2.17.2 SecretStorage-3.3.3 absl-py-2.1.0 accelerate-0.28.0 aiohttp-3.9.3 aioprometheus-23.12.0 appdirs-1.4.4 async-timeout-4.0.3 bitsandbytes-0.43.0 cffi-1.16.0 contourpy-1.2.0 cryptography-42.0.5 cycler-0.12.1 datasets-2.18.0 dill-0.3.8 docker-pycreds-0.4.0 docstring-parser-0.16 docutils-0.20.1 einops-0.7.0 fonttools-4.50.0 fsspec-2024.2.0 gitdb-4.0.11 gold-ai-olmpiad-0.1.0 grpcio-1.62.1 importlib-metadata-7.1.0 jaraco.classes-3.3.1 jaraco.context-4.3.0 jaraco.functools-4.0.0 jeepney-0.8.0 keyring-25.0.0 kiwisolver-1.4.5 lark-parser-0.12.0 markdown-3.6 markdown-it-py-3.0.0 matplotlib-3.8.3 mdurl-0.1.2 more-itertools-10.2.0 multidict-6.0.5 multiprocess-0.70.16 nh3-0.2.17 nvidia-nccl-cu12-2.19.3 orjson-3.10.0 pandas-2.2.1 peft-0.10.0 pillow-10.2.0 pkginfo-1.10.0 plotly-5.20.0 progressbar2-4.4.2 protobuf-4.25.3 pyarrow-15.0.2 pyarrow-hotfix-0.6 pycparser-2.21 pydantic-1.10.13 pyparsing-3.1.2 python-dateutil-2.9.0.post0 python-utils-3.8.2 pytz-2024.1 quantile-python-1.1 readme-renderer-43.0 requests-toolbelt-1.0.0 rfc3986-2.0.0 rich-13.7.1 scikit-learn-1.4.1.post1 sentry-sdk-1.44.0 setproctitle-1.3.3 shtab-1.7.1 six-1.16.0 smmap-5.0.1 tenacity-8.2.3 tensorboard-2.16.2 tensorboard-data-server-0.7.2 threadpoolctl-3.4.0 torch-2.2.2 torchaudio-2.2.2 torchvision-0.17.2 triton-2.2.0 trl-0.8.1 twine-5.0.0 tyro-0.7.3 tzdata-2024.1 vllm-0.2.5 wandb-0.16.5 werkzeug-3.0.1 xformers-0.0.25.post1 xxhash-3.4.1 yarl-1.9.4 zipp-3.18.1

Can you let us know precisely the commands you recommend to run? I started new conda env and it threw the above error anyway https://docs.vllm.ai/en/latest/getting_started/installation.html

error still:

_ZN2at4_ops15to_dtype_layout4callERKNS_6TensorEN3c108optionalINS5_10ScalarTypeEEENS6_INS5_6LayoutEEENS6_INS5_6DeviceEEENS6_IbEEbbNS6_INS5_12MemoryFormatEEE
brando90 commented 7 months ago

@youkaichao I followed @RylanSchaeffer advice and I still get an eror:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
outlines 0.0.37 requires pydantic>=2.0, but you have pydantic 1.10.13 which is incompatible.
Successfully installed GitPython-3.1.42 Pygments-2.17.2 SecretStorage-3.3.3 absl-py-2.1.0 accelerate-0.28.0 aiohttp-3.9.3 aioprometheus-23.12.0 appdirs-1.4.4 async-timeout-4.0.3 bitsandbytes-0.43.0 cffi-1.16.0 contourpy-1.2.0 cryptography-42.0.5 cycler-0.12.1 datasets-2.18.0 dill-0.3.8 docker-pycreds-0.4.0 docstring-parser-0.16 docutils-0.20.1 einops-0.7.0 fonttools-4.50.0 fsspec-2024.2.0 gitdb-4.0.11 gold-ai-olmpiad-0.1.0 grpcio-1.62.1 importlib-metadata-7.1.0 jaraco.classes-3.3.1 jaraco.context-4.3.0 jaraco.functools-4.0.0 jeepney-0.8.0 keyring-25.0.0 kiwisolver-1.4.5 lark-parser-0.12.0 markdown-3.6 markdown-it-py-3.0.0 matplotlib-3.8.3 mdurl-0.1.2 more-itertools-10.2.0 multidict-6.0.5 multiprocess-0.70.16 nh3-0.2.17 nvidia-nccl-cu12-2.19.3 orjson-3.10.0 pandas-2.2.1 peft-0.10.0 pillow-10.2.0 pkginfo-1.10.0 plotly-5.20.0 progressbar2-4.4.2 protobuf-4.25.3 pyarrow-15.0.2 pyarrow-hotfix-0.6 pycparser-2.21 pydantic-1.10.13 pyparsing-3.1.2 python-dateutil-2.9.0.post0 python-utils-3.8.2 pytz-2024.1 quantile-python-1.1 readme-renderer-43.0 requests-toolbelt-1.0.0 rfc3986-2.0.0 rich-13.7.1 scikit-learn-1.4.1.post1 sentry-sdk-1.44.0 setproctitle-1.3.3 shtab-1.7.1 six-1.16.0 smmap-5.0.1 tenacity-8.2.3 tensorboard-2.16.2 tensorboard-data-server-0.7.2 threadpoolctl-3.4.0 torch-2.2.2 torchaudio-2.2.2 torchvision-0.17.2 triton-2.2.0 trl-0.8.1 twine-5.0.0 tyro-0.7.3 tzdata-2024.1 vllm-0.2.5 wandb-0.16.5 werkzeug-3.0.1 xformers-0.0.25.post1 xxhash-3.4.1 yarl-1.9.4 zipp-3.18.1

Can you let us know precisely the commands you recommend to run? I started new conda env and it threw the above error anyway https://docs.vllm.ai/en/latest/getting_started/installation.html

error still:

_ZN2at4_ops15to_dtype_layout4callERKNS_6TensorEN3c108optionalINS5_10ScalarTypeEEENS6_INS5_6LayoutEEENS6_INS5_6DeviceEEENS6_IbEEbbNS6_INS5_12MemoryFormatEEE

ok it seems this is sensitive to python version. You have to do 3.9. Then the link by rylan works.

Code:

conda create -n myenv python=3.9 -y
conda activate myenv
# needed in this order for vllm to work
pip install vllm 
# optional
pip install -e ~/your_library/setup.py

ref: https://docs.vllm.ai/en/latest/getting_started/installation.html

brando90 commented 7 months ago

@youkaichao what version of pytorch is supported then?

brando90 commented 7 months ago

is it pytorch 2.1.2?

youkaichao commented 7 months ago

If you build vllm from source, it supports (requires) pytorch 2.2 now.

brando90 commented 7 months ago

If you build vllm from source, it supports (requires) pytorch 2.2 now.

I'm using pip though.

youkaichao commented 7 months ago

pip install vllm directly, still uses pytorch 2.1 . stay tuned when we make the next release.

brando90 commented 7 months ago

pip install vllm directly, still uses pytorch 2.1 . stay tuned when we make the next release.

awesome! any estimate? Thank you!

brando90 commented 7 months ago

@youkaichao since the versions of python I am using are fragile due to the current vllm (or one I use) only works with pytorch 2.1, I was wondering, what hugging face and accelerate version do we need without breaking vllm?

Need to debug this but I think this should work:

  # for pytorch see doc string at the top of file
    install_requires=[
        'dill',
        'networkx>=2.5',
        'scipy',
        'scikit-learn',
        'lark-parser',
        'tensorboard',
        'pandas',
        'progressbar2',
        'requests',
        'aiohttp',
        'numpy',
        'plotly',
        'wandb',
        'matplotlib',
        # 'statsmodels'
        # 'statsmodels==0.12.2'
        # 'statsmodels==0.13.5'
        # - later check why we are not installing it...
        # 'seaborn'
        # 'nltk'
        'twine',

        'torch==2.1.2',  # 2.2 not supported due to vllm see: https://github.com/vllm-project/vllm/issues/2747
        # 'torchvision',
        # 'torchaudio',

        # 'fairseq',

        # 'trl',
        'transformers==4.39.2', # my gold-ai-olympiad project uses 4.39.2
        'accelerate==0.29.2',
        # 'peft',

        'datasets==2.18.0',  # 2.18.0
        'bitsandbytes== 0.43.0',
        # 'einops',

        'vllm==0.4.0.post1', # my gold-ai-olympiad project uses 0.4.0.post1 ref: https://github.com/vllm-project/vllm/issues/2747
    ]
)

and fyi:

(snap_cluster_setup) brando9@skampere1~/lean4ai $ pip list
Package                   Version
------------------------- ------------
absl-py                   2.1.0
accelerate                0.29.2
aiohttp                   3.9.3
aiosignal                 1.3.1
annotated-types           0.6.0
anyio                     4.3.0
appdirs                   1.4.4
async-timeout             4.0.3
attrs                     23.2.0
backports.tarfile         1.0.0
bitsandbytes              0.43.0
certifi                   2024.2.2
cffi                      1.16.0
charset-normalizer        3.3.2
click                     8.1.7
cloudpickle               3.0.0
cmake                     3.29.0.1
contourpy                 1.2.1
cryptography              42.0.5
cycler                    0.12.1
datasets                  2.18.0
dill                      0.3.8
diskcache                 5.6.3
docker-pycreds            0.4.0
docutils                  0.20.1
exceptiongroup            1.2.0
fastapi                   0.110.1
filelock                  3.13.3
fonttools                 4.51.0
frozenlist                1.4.1
fsspec                    2024.2.0
gitdb                     4.0.11
GitPython                 3.1.43
grpcio                    1.62.1
h11                       0.14.0
httptools                 0.6.1
huggingface-hub           0.22.2
idna                      3.6
importlib_metadata        7.1.0
importlib_resources       6.4.0
interegular               0.3.3
jaraco.classes            3.4.0
jaraco.context            5.3.0
jaraco.functools          4.0.0
jeepney                   0.8.0
Jinja2                    3.1.3
joblib                    1.4.0
jsonschema                4.21.1
jsonschema-specifications 2023.12.1
keyring                   25.1.0
kiwisolver                1.4.5
lark                      1.1.9
lark-parser               0.12.0
llvmlite                  0.42.0
Markdown                  3.6
markdown-it-py            3.0.0
MarkupSafe                2.1.5
matplotlib                3.8.4
mdurl                     0.1.2
more-itertools            10.2.0
mpmath                    1.3.0
msgpack                   1.0.8
multidict                 6.0.5
multiprocess              0.70.16
nest-asyncio              1.6.0
networkx                  3.2.1
nh3                       0.2.17
ninja                     1.11.1.1
numba                     0.59.1
numpy                     1.26.4
nvidia-cublas-cu12        12.1.3.1
nvidia-cuda-cupti-cu12    12.1.105
nvidia-cuda-nvrtc-cu12    12.1.105
nvidia-cuda-runtime-cu12  12.1.105
nvidia-cudnn-cu12         8.9.2.26
nvidia-cufft-cu12         11.0.2.54
nvidia-curand-cu12        10.3.2.106
nvidia-cusolver-cu12      11.4.5.107
nvidia-cusparse-cu12      12.1.0.106
nvidia-nccl-cu12          2.18.1
nvidia-nvjitlink-cu12     12.4.127
nvidia-nvtx-cu12          12.1.105
outlines                  0.0.34
packaging                 24.0
pandas                    2.2.1
pillow                    10.3.0
pip                       24.0
pkginfo                   1.10.0
plotly                    5.20.0
progressbar2              4.4.2
prometheus_client         0.20.0
protobuf                  4.25.3
psutil                    5.9.8
py-cpuinfo                9.0.0
pyarrow                   15.0.2
pyarrow-hotfix            0.6
pycparser                 2.22
pydantic                  2.6.4
pydantic_core             2.16.3
Pygments                  2.17.2
pynvml                    11.5.0
pyparsing                 3.1.2
python-dateutil           2.9.0.post0
python-dotenv             1.0.1
python-utils              3.8.2
pytz                      2024.1
PyYAML                    6.0.1
ray                       2.10.0
readme_renderer           43.0
referencing               0.34.0
regex                     2023.12.25
requests                  2.31.0
requests-toolbelt         1.0.0
rfc3986                   2.0.0
rich                      13.7.1
rpds-py                   0.18.0
safetensors               0.4.2
scikit-learn              1.4.1.post1
scipy                     1.13.0
SecretStorage             3.3.3
sentencepiece             0.2.0
sentry-sdk                1.44.1
setproctitle              1.3.3
setuptools                68.2.2
six                       1.16.0
smmap                     5.0.1
sniffio                   1.3.1
starlette                 0.37.2
sympy                     1.12
tenacity                  8.2.3
tensorboard               2.16.2
tensorboard-data-server   0.7.2
threadpoolctl             3.4.0
tiktoken                  0.6.0
tokenizers                0.15.2
torch                     2.1.2
tqdm                      4.66.2
transformers              4.39.2
triton                    2.1.0
twine                     5.0.0
typing_extensions         4.11.0
tzdata                    2024.1
urllib3                   2.2.1
uvicorn                   0.29.0
uvloop                    0.19.0
vllm                      0.4.0.post1
wandb                     0.16.6
watchfiles                0.21.0
websockets                12.0
Werkzeug                  3.0.2
wheel                     0.41.2
xformers                  0.0.23.post1
xxhash                    3.4.1
yarl                      1.9.4
zipp                      3.18.1

For flash attention I have these comments

       # # ampere
        # 'dspy-ai',
        # # 'torch==2.1.2+cu118',  # 2.2 not supported due to vllm see: https://github.com/vllm-project/vllm/issues/2747
        # # 'torch==2.1.2',  # 2.2 not supported due to vllm see: https://github.com/vllm-project/vllm/issues/2747
        # # 'torch==2.2.1',  # 2.2 not supported due to vllm see: https://github.com/vllm-project/vllm/issues/2747
        # 'torch==2.2.1',  # 2.2 not supported due to vllm see: https://github.com/vllm-project/vllm/issues/2747
        # # 'torchvision',
        # # 'torchaudio',
        # # 'trl',
        # # 'transformers==4.39.2',
        # 'transformers>=4.40',
        # 'accelerate==0.29.2',
        # # 'peft',
        # # 'datasets==2.18.0', 
        # 'datasets==2.14.7',  
        # 'evaluate==0.4.1', 
        # 'bitsandbytes== 0.43.0',
        # 'einops',
        # 'flash-attn>=2.5.8',
        # 'vllm==0.4.1', # my gold-ai-olympiad project uses 0.4.0.post1 ref: https://github.com/vllm-project/vllm/issues/2747
        # # pip install -q -U google-generativeai
youkaichao commented 7 months ago

please install vllm in a fresh new environment, then you don't need to care about this manually.