Consider using a pod for Ray processes

[kklin]

Looking at the docs, it looks like ray start starts multiple processes (e.g. an object store, local scheduler, and more).

Running these processes in separate containers might make the system easier to use. The two things I'm thinking of are:

• Logging. Logs don't seem to show up in kelda logs worker right now. I suspect that they go to disk. But even if we redirected all the logs to stdout, they'd be confusingly intertwined since there are multiple processes logging.
• Crash recovery. Right now, if one of the processes die, it won't be restarted (unless Ray has a process-watcher). Pulling the processes out into containers would allow Kubernetes to restart the container if it dies.

[kklin]

This wouldn't happen anytime soon I think, but might be a useful thought experiment.

[luise]

Ah, yeah, that definitely seems helpful. I wonder if that's something we would eventually do, or if the Ray team themselves will eventually change their Docker images to be more "container native".