search

lyogavin / airllm

AirLLM 70B inference with single 4GB GPU
Apache License 2.0
4.77k stars 384 forks source link

AirLLM: Support for DirectML #108

Open vegax87 opened 9 months ago

vegax87 commented 9 months ago

DirectML can run any LLM on any GPU that supports DX12:

if opt.use_dml:
        import torch_directml
        device = torch_directml.device(torch_directml.default_device())
    else:
        device = torch.device('cuda' if opt.cuda else 'cpu')