In order to be able to access objects after spark executors shutdown, we can assign the ownership to RayDPSparkMaster. But for now, we cannot cleanup the reference, so objects created such way cannot be free. This will be a memory issue when dealing with large amount of data.
This PR adds a function to delete such references. And to be able to do this, we use object.hex() (string repr of object id) as key. This function needs to be called manually by user when user finish using the object.
In order to be able to access objects after spark executors shutdown, we can assign the ownership to RayDPSparkMaster. But for now, we cannot cleanup the reference, so objects created such way cannot be free. This will be a memory issue when dealing with large amount of data.
This PR adds a function to delete such references. And to be able to do this, we use object.hex() (string repr of object id) as key. This function needs to be called manually by user when user finish using the object.