Open bradday4 opened 3 years ago
I don't have a ray server around, unfortunately. That said, part of me is also wondering to what extent we want to benchmark this. Having a benchmark that tells us this change is meaningful might be enough, mainly because we might need feedback from other users to get a good picture of what's important to improve/focus on.
On my end, this looks green to start working on a PR. 👍
Sounds good. I'll start working on a PR then. Do you want the PR to go back to Main? Or should there be a dev branch to submit PR's to. I noticed I missed deleting an unused import last time that caused a build failure on your end.
Feel free to push to main. You might want to pull first though, I cleaned up the repo with regards to docs in between your PR and this new one.
So I did some testing and it seems as though adding support for ray should be relatively straight forward. I've only tested this locally as I don't have a remote machine to set up on right now. I really don't know how the
data
list will be appended to in a remote context. Something definitely worth investigating I think. To which, if you have an extra ray server laying around and don't mind sharing it I'd be more than happy to test it out.The following will enable joblib to work with ray locally.
From the shell
At the top of
demo.py
And at the beginning of
Runner._run
This should leave the memo package decoupled from ray so it's optional for the user. We could also catch the import error and default back to joblibs
loky
but my feeling is that if someone passes the backend as ray then failing would be better then raising a warning and continuing running locally. Your thoughts?