josevalim
commented
3 months ago Another reason why we use a name is because we also use as a key in persistent_term to store serving information. We also need a group name to lookup things in pg. I am not sure the API allows us to return a group for a PID?
If you need to dynamically manage servings, you can always defined a pool of static names, such as Pool1, Pool2, Pool3, etc. As long as they are bound, you should be fine.
Currently servings are looked up by name. This is fine since serving topology so far has been static. However, I've been working on dynamically managing servings, and need more fine grained control as to which serving a task is run against. Hence, I propose that batched runs can be done against a pid.
This is trivial for local runs: we eliminate looking up the name, and run as usual. For distributed runs, we validate if the pid exists in the PG instead of selecting one.
A concern for supporting this for distributed runs is that this would bypass all load balancing.