search

astronomer / astro-provider-ray

This provider contains operators, decorators and triggers to send a ray job from an airflow task
https://astronomer.github.io/astro-provider-ray/
Apache License 2.0
12 stars 3 forks source link

Add documentation for using the RayProvider for Cloud Providers #76

Open CaliWee opened 1 month ago

CaliWee commented 1 month ago

I am using Ray on GKE in Google Cloud. How can I specify where Ray is running using the RayOperators?

Currently, we use the GKEOperator which allows you to specify things like project_id, and cluster_name - https://cloud.google.com/composer/docs/using-gke-operator#launch_workloads_in_the_cluster

CaliWee commented 1 month ago

After diving in a bit further, it looks like it could be possible by being able to configure these attributes of the RayHook - https://github.com/astronomer/astro-provider-ray/blob/main/ray_provider/hooks/ray.py#L104-L106, but I haven't been able to find docs on _get_field and it doesn't seem like these fields are exposed to the user at the moment from higher level classes like the SubmitJobOperator