Our endpoint system (eps) that manages and secures the communication between different actors in the IRIS connect ecosystem. Think of it as a distributed service mesh router as well as a decentralized message broker. Still evolving, use with caution.
Currently, if we define e.g. grpc_server, grpc_clientand jsonrpc_client channels for a given server (in that particular order) and send a message to that server, the grpc_client will discover that a message addressed to the server can be delivered by it via the grpc_server channel. The client will then resubmit the message to the grpc_server and that server will again pass it into the message broker. The the message will then circulate indefinitely...
Solution
Channels should be tagged as local to indicate they can deliver messages addressed for the server itself. Those messages should not be passed to the other channels to avoid redelivering them again and again to the same channel.
In addition, a message broker should keep a data structure with the message IDs that are being processed, and reject any message that is already being processed. As circular delivery loops could theoretically also occur in a multi-server setup (e.g. server A delivers to B, B delivers to C, C delivers to A) we should ensure this behavior cannot cause an infinite delivery loop.
Other Things
Validate that all file handles are being correctly closed as an "out of file handlers" error was observed when this bug occurred.
adewes
commented
3 years ago
Currently, if we define e.g.
grpc_server,grpc_clientandjsonrpc_clientchannels for a given server (in that particular order) and send a message to that server, thegrpc_clientwill discover that a message addressed to the server can be delivered by it via thegrpc_serverchannel. The client will then resubmit the message to thegrpc_serverand that server will again pass it into the message broker. The the message will then circulate indefinitely...Solution
Channels should be tagged as
localto indicate they can deliver messages addressed for the server itself. Those messages should not be passed to the other channels to avoid redelivering them again and again to the same channel.In addition, a message broker should keep a data structure with the message IDs that are being processed, and reject any message that is already being processed. As circular delivery loops could theoretically also occur in a multi-server setup (e.g. server A delivers to B, B delivers to C, C delivers to A) we should ensure this behavior cannot cause an infinite delivery loop.
Other Things
Validate that all file handles are being correctly closed as an "out of file handlers" error was observed when this bug occurred.