ModuleNotFoundError: No module named 'sglang.srt.managers.router'

[LiuZhetan]

I try to use preble to deploy a model by sglang, but get an error:

$ preble run -port 6666 -model /workspace/LLMs/Qwen2-7B-Instruct
Traceback (most recent call last):
  File "/usr/local/bin/preble", line 5, in <module>
    from preble.server.server import main
  File "/usr/local/lib/python3.10/dist-packages/preble/server/server.py", line 20, in <module>
    from sglang.srt.managers.router.model_runner import GPUConfig
ModuleNotFoundError: No module named 'sglang.srt.managers.router'

I found that the built-in sglang version is v0.1.16, but there is no GPUConfig class in the official v0.1.16 code, and I did not find it in the previous version either.

So did the author modify the original sglang code?

My environment:

Python 3.10.14
sglang 0.3.0
torch 2.4.0

[vikranth22446]

I did modify the original sglang code for experiments & for providing profiling numbers. You would have to install from source if you want to run it.

The flashinfer version on the main branch I believe might also be from an older version v0.3 as well. I have some profiling numbers on the latest version, but haven't updated the main branch.