Require a channel when sending messages in the topology

[josevalim]

See https://arxiv.org/pdf/1802.02652.pdf.

[bitwalker]

Since a channel will be required, how do you plan to expose that configuration? Is the plan to point people to Partisan or provide an alternative as part of this project? Strictly speaking, this is really more of an optional thing rather than required, since the default Erlang distribution channel is sufficient for small clusters - I think it's only in large clusters where congestion becomes a problem.

[josevalim]

We want to change the Firenest.Topology API so that a channel is expected as argument when sending messages. This will allow us to use Partisan efficiently or allow someone to implement better multiplexing than the disterl that uses a single connection.

[bitwalker]

That makes sense, but what is the channel abstraction? As far as I'm aware, there is nothing for distributed Erlang, it's just baked into the kernel modules (primarily :rpc) - is Firenest planning on exposing a behavior or protocol for other libraries to standardize on, or is there one already available that I'm not aware of? I didn't see anything like that in Partisan, but I skimmed through, so I may have missed it.

[josevalim]

The name "channel" is confusing in our context, so we will need something better. The idea though is to tag messages and we provide ordering guarantee only for messages with the same tag. This allows us to open up multiple connections between nodes. More info on section 3.3.2 in the paper.

[bitwalker]

I think it's good terminology, but yeah, in the context of Phoenix it will certainly be confusing.

In any case, I think I get it now, in Firenest.Topology.Erlang, the channel tag would be ignored, but for a topology based on Partisan, the tag would be passed through to the Partisan API, is that correct? I'm assuming the channel tag would be optional when sending messages using the Firenest API, but required by Firenest.Topology implementations, since that appears to be how Partisan works as well (non-tagged messages just use a default channel).

[josevalim]

Yes, precisely. :+1:

[Qqwy]

Some other names instead of "Channel":

Accurate but probably bad because of its use in different (but related) contexts:

• Socket
• Pipe

Maybe better; all of these attempt to capture the 'messages are ordered when having(/being on) the same X':

• Plane
• Level
• Line
• Track
• Route

[josevalim]

I think I would call it a partition and probably force them to be integers. If we don’t want to impose integers, then I would say the argument is always hashed. --

José Valimwww.plataformatec.com.br http://www.plataformatec.com.br/Founder and Director of R&D

[Qqwy]

I think I would call it a partition and probably force them to be integers.

Having now read the Partisan paper thoroughly, I disagree with both of these proposals because:

• The name 'Partition' already is used in the meaning of 'network (/node cluster) partition', which is a common occurence in exactly the mental context we'd use Partisan/Firenest in, so we'd have two things with the same name but drastically different meanings within the same mental context.
• In the paper, the multiple Partisan Channels are used to disambugate between message types: 'normal' process messages use the default channel, while 'meta' messages (like the ones that perform synchronization of the Riak database ring state or that perform anti-entropy traffic to optimize the network topology. See paper § 4.2) use their own Partisan Channel. Using unnamed integers for these seems like a bad choice because this would result in magic numbers: 33 does not mean anything to a reader of the code, while something like :riak_state_gossip does. So it to me makes sense to at least allow the use of atoms to identify Partisan Channels (or their equivalent in Firenest).

[josevalim]

@Qqwy right. so we can just allow them to be any term, which we will hash and send in the available number of underlying connections.

[michalmuskala]

I'm not sure we want to do hashing - I think the goal is to send a particular message on a particular connection and not just have a pool where you multiplex messages.