Closed glennmoy closed 10 months ago
I've narrowed it down to the changes in https://github.com/beacon-biosignals/Ray.jl/commit/caddc0e9d0586ed69e52a8a5593dba1226833b93
I can confirm that https://github.com/beacon-biosignals/Ray.jl/commit/25b0670f9fdad02e43777b87ced000c783736e58 is successful while the previous commit https://github.com/beacon-biosignals/Ray.jl/commit/80a7b795b906b9be78cb2b1bf5b3f198426821f7 fails
I have no idea why slurping the return of parse_ray_args_from_raylet
would cause this.
# we use session_dir here instead of logs_dir since logs_dir can be set to
# "" to disable file logging without using env var
- args = parse_ray_args_from_raylet_out(session_dir)
- gcs_address = args[3]
- node_ip_address = args[4]
+ raylet, store, gcs_address, node_ip_address, node_port = parse_ray_args_from_raylet_out(session_dir)
+ # gcs_address = args[3]
+ # node_ip_address = args[4]
# @info "args: $args"
This is running on a in-house ray cluster on a custom built docker image.
Sanitised logs: