search

oap-project / raydp

RayDP provides simple APIs for running Spark on Ray and integrating Spark with AI libraries.
Apache License 2.0
293 stars 66 forks source link

Upgrade raydp to be compatible with ray 2.0.0 #267

Closed kira-lin closed 1 year ago

pang-wu commented 1 year ago

We might also want to migrate off the usage of ray.worker.global_worker.node, they are marked as deprecated. But right now I didn't find an alternative in the public API..

kira-lin commented 1 year ago

@pang-wu Yes, I considered to replace it. But the key point is we have to use _deserialize_and_register in global_worker.core_worker now.

pang-wu commented 1 year ago

@pang-wu Yes, I considered to replace it. But the key point is we have to use _deserialize_and_register in global_worker.core_worker now.

@kira-lin Isn't that still using ray.worker.*? I think we might need to migrate off from using the whole ray._private.* and rely on public API - or we plan to stay with private for a while until Ray folks figure out the public API?

kira-lin commented 1 year ago

Yes, we'll need to make some upstream changes. For now we can't only use public APIs, we would not be able to call from_spark in that case.

pang-wu commented 1 year ago

gotcha, thanks for the prompt responses!

carsonwang commented 1 year ago

Thanks @kira-lin ! Merging this.