search

microsoft / ray-on-aml

Turning AML compute into Ray cluster
Other
77 stars 12 forks source link

Bump version to ray 1.13.0 #15

Closed edgBR closed 1 year ago

edgBR commented 2 years ago

The current code is not installing dependencies properly in the cluster.

When doing:

from azureml.core import Workspace, Experiment, Environment,ScriptRunConfig
# from azureml.widgets import RunDetails
from azureml.core.compute import ComputeTarget, AmlCompute
from azureml.core.compute_target import ComputeTargetException
from azureml.core.environment import Environment
from ray_on_aml.core import Ray_On_AML
import time

ws = Workspace.from_config()
ray_on_aml = Ray_On_AML(ws=ws, 
                        compute_cluster ="ray-final-test", 
                        maxnode=max_cluster_nodes,  
ray_on_aml.getRay()

Runtime error appears:

RuntimeError: Version mismatch: The cluster was started with:
    Ray: 1.13.0
    Python: 3.8.5
This process on node 10.0.0.5 was started with:
    Ray: 1.12.0
    Python: 3.8.5
james-tn commented 1 year ago

Your CI has ray version 1.13 while the ray-on-aml cluster has version 1.12 (by default). You can fix ray version in either environment. It's better to use Ray client mode (ci_is_head=False) in getRay as the new ray client can work just like you're on head node

james-tn commented 1 year ago

Hey, check out the new version. You basically control which ray version you want to run .