[Feature]: Multi-node tasks IP discovery

[g-ananth]

Problem

Feature request

I want to provision a collection of containers that need to collectively process a workload. The workload could be a streaming/batch workload. One of the aspects that is needed as part of the provisioning phase of the workload is that all of the workers need to know that ip addresses/DNS names of all the peers. (There is no master IP and passive workers construct in the framework).

Solution

Ideally the solution could provide the following:

• An environment variable that is a JSON string that can be parsed into a dictionary
• The key part of the dictionary is the index id of the fleet instance 0 being the master)
• The value part is the DNS host name (ideally ) as that could take care of the retries ??

Workaround

Not aware of the workarounds.

Would you like to help us implement this feature by sending a PR?

Yes

[peterschmidt85]

Thank you, @g-ananth, for reporting this. We're looking into it. Two questions:

1. Are you using a specific distributed framework? If so, which one?
2. How urgent is this issue? By when would it be critical to have it working?

[g-ananth]

Thanks for offering to look into it.

for [1] : Currently looking at bytewax (bytewax does provide k8s based constructs but want to avoid the k8s way of spawning workloads as our workflows are dynamically crafted based on the context). Here is a link that might help in more context. We have a separate orcehstrator that will be using the dstack python API to spawn the workloads. for [2]: It is fairly core for us in evaluating dstack. Having said that I reckon open source has its pace. Happy to chime in and contribute if there is some guidance on how a beginner contributor can help in this ticket. We do have skypilot in the stack but it is more for ray ecosystem (and implicitly spins up ray underneath). We are looking at use cases that are not ray enabled but rather the rest of the patterns which is where dstack comes into picture.

[r4victor]

@g-ananth, we're going to prioritize this feature now and try to include it in the release this or next week.

Regarding the solution, in general, the nodes don't have DNS names, so we'll expose nodes' internal IPs. You also suggested a JSON encoding but I think we should aim for a simpler format like a string of IPs delimited by newlines so that it can be easily processed in bash, for example: "10.0.0.1\n10.0.0.2\n10.0.0.3\n10.0.0.3". Please confirm if it works for your.

[g-ananth]

@r4victor Yes that will work, thank you. Agree that a single string is better processable in bash.

[r4victor]

@g-ananth, added a new DSTACK_NODES_IPS env variable. It'll be included in the next release coming this week.