capnproto / pycapnp

Cap'n Proto serialization/RPC system - Python bindings
BSD 2-Clause "Simplified" License
458 stars 125 forks source link

"Randomly" missing RPC calls #318

Open simon-staal opened 1 year ago

simon-staal commented 1 year ago

I'm working on a two-way client server RPC system using capnp and asyncio. I've set everything up following the docs / examples, and everything works most of the time, but every once in a while , either the server or client will try calling a method via the other side's interface which is never received (or processed) on the callee side, i.e. logs on the callee side indicating the method was called are not being produced, and the await call on the caller side never resolves. This happens most notably for "heartbeat" messages which are sent from clients to the server at 5 second intervals, but not exclusively.

I'm currently testing everything locally, so dropped packets should be impossible, but even if I was testing across an actual network connection, TCP's retransmission protocols should make this "impossible" as well, which is why I believe this to be a capnproto issue.

Apologies for this request being so vague, the issue is very difficult to reproduce (hence the "randomly"), sometimes it doesn't happen for 30 mins+, making it quite difficult to gather more details (such as if the myreader method on the callee side is passing the bytes received on the socket to the capnp client). My schema / implementations are also a bit long to include in an issue like this, although please let me know if there's anything in particular you'd like me to add. I noticed that direct asyncio integration will be provided in the upcoming major release, can I expect this to solve the issue, or is it more likely I'm doing something wrong on my end? The current workaround I'm using is to wrap any requests in asyncio.wait_fors, but this is less than ideal for a number of reasons.