[Feature] Whether pytorch backend is supported on Windows?

[eeyrw]

Motivation

I tried to deploy InterlVL-1B by LMDeploy which said this model is not supported by turbomind so I added ‘--backend pytorch’ but finally got such kind of error:

(dl_venv) PS xxxxxxxxxxxx> lmdeploy serve gradio "F:\InternVL2-1B" --server-name localhost --server-port 6006 --backend pytorch
2024-10-26 12:29:04,466 - lmdeploy - INFO - builder.py:55 - matching vision model: InternVLVisionModel
FlashAttention2 is not installed.
2024-10-26 12:29:07,607 - lmdeploy - INFO - internvl.py:113 - using InternVL-Chat-V1-5 vision preprocess
2024-10-26 12:29:07,623 - lmdeploy - INFO - engine.py:135 - start ImageEncoder._forward_loop
2024-10-26 12:29:07,623 - lmdeploy - INFO - async_engine.py:142 - input backend=pytorch, backend_config=PytorchEngineConfig(dtype='auto', tp=1, session_len=8192, max_batch_size=128, cache_max_entry_count=0.8, prefill_interval=16, block_size=64, num_cpu_blocks=0, num_gpu_blocks=0, adapters=None, max_prefill_token_num=8192, thread_safe=True, enable_prefix_caching=False, device_type='cuda', eager_mode=False, custom_module_map=None, download_dir=None, revision=None)
2024-10-26 12:29:07,623 - lmdeploy - INFO - async_engine.py:144 - input chat_template_config=None
2024-10-26 12:29:07,638 - lmdeploy - INFO - async_engine.py:154 - updated chat_template_onfig=ChatTemplateConfig(model_name='internvl2-internlm2', system=None, meta_instruction=None, eosys=None, user=None, eoh=None, assistant=None, eoa=None, separator=None, capability=None, stop_words=None)
2024-10-26 12:29:07,638 - lmdeploy - INFO - __init__.py:98 - Checking environment for PyTorch Engine.
2024-10-26 12:29:07,732 - lmdeploy - ERROR - __init__.py:17 - ModuleNotFoundError: No module named 'triton'
2024-10-26 12:29:07,732 - lmdeploy - ERROR - __init__.py:18 - <Triton> test failed!
Please ensure it has been installed correctly.

Triton does not support Windows. So does it mean pytorch backend of LMDeploy is totally not supported on Windows? Is there any workaround for this?

Related resources

No response

Additional context

No response

[wjn922]

We only conduct testing on Linux. Please consult the LMDeploy team (https://github.com/InternLM/lmdeploy) for this issue.

[eeyrw]

Sure. I just did not notice this repo is for InternVL rather than LMDeploy.

For those who care this: https://github.com/InternLM/lmdeploy/issues/2684