search

dstackai / dstack

dstack is an easy-to-use and flexible container orchestrator for running AI workloads in any cloud or data center.
https://dstack.ai
Mozilla Public License 2.0
1.19k stars 87 forks source link

[Bug]: RunPod backend does not respect configured regions #1348

Open r4victor opened 1 week ago

r4victor commented 1 week ago

Steps to reproduce

  1. Specify regions property for runpod backend:
type: runpod
  regions: [EU-RO-1]
  creds:
    type: api_key
    api_key: my_api_key

  1. Observe that offers from other regions are included:

    ✗ dstack run . -b runpod
Configuration          .dstack.yml                    
Project                main                           
User                   admin                          
Pool                   default-pool                   
Min resources          2..xCPU, 8GB.., 100GB.. (disk) 
Max price              -                              
Max duration           6h                             
Spot policy            on-demand                      
Retry policy           no                             
Creation policy        reuse-or-create                
Termination policy     destroy-after-idle             
Termination idle time  5m                             

#  BACKEND  REGION    INSTANCE                RESOURCES              SPOT  PRICE   
1  runpod   CA-MTL-1  NVIDIA RTX A4000        9xCPU, 50GB,           no    $0.19   
                                           1xRTXA4000 (16GB),                   
                                           100.0GB (disk)                       
2  runpod   EU-RO-1   NVIDIA RTX 4000 Ada     9xCPU, 50GB,           no    $0.21   
                   Generation              1xRTX4000Ada (20GB),                 
                                           100.0GB (disk)                       
3  runpod   EU-RO-1   NVIDIA RTX A4500        12xCPU, 31GB,          no    $0.21   
                                           1xRTXA4500 (20GB),                   
                                           100.0GB (disk)                       
...

I don't see that RunpodCompute uses config.regions anywhere, so this is probably the issue:

https://github.com/dstackai/dstack/blob/9d1cf0ec453920e6f8952e664c327dde146a07e2/src/dstack/_internal/core/backends/runpod/compute.py#L35-L39

Actual behaviour

No response

Expected behaviour

No response

dstack version

master

Server logs

No response

Additional information

No response

r4victor commented 1 week ago

This also seems to be the case with cudo, tensordock, vastai backends.