epi-project / brane

Programmable Orchestration of Applications and Networking
Apache License 2.0
10 stars 8 forks source link

Implement data management API for worker nodes #186

Open DanielVoogsgerd opened 3 weeks ago

DanielVoogsgerd commented 3 weeks ago

Moving data into a worker node is often quite a hassle. We have to copy data into the workers data directory, after which we have to change the absolute path to match the new location.

I suggest the following changes:

@lut99, what do you think?

Lut99 commented 2 weeks ago

Good point!

  1. Yes plz, even I'm always hazy on which does which;
  2. That's OK, provided that nothing in the code directly uses assets anymore OR any part that uses it has its code updated to canonicalize the path before mounting it to a Docker volume if it doesn't already (Docker requires absolute paths).
  3. I'm against a brane data push; it's not the scientist's job to make data available on the worker node simply because it requires updating the policy on the worker. Instead, we can make a branectl data push (which is the worker-side tool, after all). Longer-term, I have in my mind to either have central host a package-like repository for data to which scientists can push (but necessarily isn't shielded by policy); or to make an optional flag for workers to allow "guest data".

That aside, I agree it's a hassle. Something like branectl data push is most definitely a very nice option (or perhaps branectl data build, to mirror brane's operation but for worker managers).