search

n0-computer / iroh

A toolkit for building distributed applications
https://iroh.computer
Apache License 2.0
2.27k stars 145 forks source link

Backpressure handling when derp traffic is faster than local processing #1977

Open rklaehn opened 7 months ago

rklaehn commented 7 months ago

@ppodolsky hit this issue:

2024-01-24T20:09:53.623716Z  WARN magicsock:derp-actor:active-derp: dropping received DERP packet: "Full(..)" me=... url=https://euw1-1.derp.iroh.network./
2024-01-24T20:09:53.623791Z  WARN magicsock:derp-actor:active-derp: dropping received DERP packet: "Full(..)" me=... url=https://euw1-1.derp.iroh.network./
2024-01-24T20:09:53.623811Z  WARN magicsock:derp-actor:active-derp: dropping received DERP packet: "Full(..)" me=... url=https://euw1-1.derp.iroh.network./
2024-01-24T20:09:53.623827Z  WARN magicsock:derp-actor:active-derp: dropping received DERP packet: "Full(..)" me=... url=https://euw1-1.derp.iroh.network./
2024-01-24T20:09:53.623844Z  WARN magicsock:derp-actor:active-derp: dropping received DERP packet: "Full(..)" me=... url=https://euw1-1.derp.iroh.network./
2024-01-24T20:09:53.623860Z  WARN magicsock:derp-actor:active-derp: dropping received DERP packet: "Full(..)" me=... url=https://euw1-1.derp.iroh.network./
2024-01-24T20:09:53.624204Z  WARN magicsock:derp-actor:active-derp: dropping received DERP packet: "Full(..)" me=... url=https://euw1-1.derp.iroh.network./
2024-01-24T20:09:53.624227Z  WARN magicsock:derp-actor:active-derp: dropping received DERP packet: "Full(..)" me=... url=https://euw1-1.derp.iroh.network./
2024-01-24T20:09:53.624250Z  WARN magicsock:derp-actor:active-derp: dropping received DERP packet: "Full(..)" me=... url=https://euw1-1.derp.iroh.network./
2024-01-24T20:09:53.624276Z  WARN magicsock:derp-actor:active-derp: dropping received DERP packet: "Full(..)" me=... url=https://euw1-1.derp.iroh.network.

This seems to drop packets coming from the DERP relay when local processing can not keep up. Since the connection to the relay is a TCP connection, could we just use the TCP backpressure instead?

flub commented 6 months ago

question is whether that really is any better. It is only incidental that the derper connection is TCP (and a little unfortunate). As far as quinn is concerned it is sending datagrams somewhere. The inside of iroh is just another router along that path, things go as far as they can and are then dropped when the buffer is full. That is how UDP works.

If we use TCP's backpressure I think we'll confuse things more and move the problem around. Probably next is the derper who would have to either drop the packets or slow down the flow from the derper channel to sending node. But that it can't do, because it needs to keep that flowing so that DISCO packets still get through.

So, my take is that this is fine. Maybe we should trim down the logging to be somewhat more reasonable though.

dignifiedquire commented 6 months ago

lets at least reduce the logs for now