Closed hkunzhe closed 1 month ago
autogptq has conflicts with sglang in the docker image:
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
Therefore, we need to separate these two imports.
autogptq has conflicts with sglang in the docker image:
Therefore, we need to separate these two imports.