search

LLMServe / DistServe

Disaggregated serving system for Large Language Models (LLMs).
Apache License 2.0
122 stars 9 forks source link

missing SimulatorConfig.profiler_data_path when runing offline.py #7

Closed KylinC closed 2 weeks ago

KylinC commented 3 weeks ago

missing SimulatorConfig.profiler_data_path when runing offline.py

llx-08 commented 2 weeks ago

If your intention is merely to run offline.py, you can manually modify the class OfflineLLM in the /distserve/llm.py file by adding a SimulatorConfig instance and setting is_simulator_mode to False. like this

       # modified
       simulator_config = SimulatorConfig(is_simulator_mode=False,
                                           profiler_data_path='',
                                           gpu_mem_size_gb=70)

        self.engine = LLMEngine(
            model_config,
            disagg_parallel_config,
            cache_config,
            context_sched_config,
            decoding_sched_config,
            simulator_config
        )
PKUFlyingPig commented 2 weeks ago

This bug has been fixed in PR(#11), please try again.

interestingLSY commented 2 weeks ago

Closing this issue. Reopen it if you encountered any further issues.