Bug: sometimes hit rate limit error when we haven't really hit the rate limit

[orbitalturtle]

Describe the bug

Sometimes we hit Peer: 02ad2e51564b763f65c934d67aa6528ad8238236691457c65b34fc49f843ec0662 hit rate limit, dropping incoming onion message when we haven't connected to the peer before but are getting an onion message from them.

I need to refresh on how the rate limiter works, but seems like we shouldn't assume that the node has hit the rate limit just because the rate limiter isn't aware of the node yet. In this case, it's just that the peer connect notification doesn't come through until after we receive the onion message.

To Reproduce

I've been using our lndk-cli pay-offer command, which is still a PR ATM. This error comes up occasionally in regtest when we've sent the invoice request to the offer creator and are waiting for the invoice to come back. If we haven't connected to the node that's responding with the invoice, we sometimes hit the rate limit error:

2024-02-20T18:10:27.463355343-06:00 INFO lndk::onion_messenger - Sending outgoing onion message to 02a5c20878dfd77120669dda88a9953f476fef1f8094550901f386dd378a7aed1f.
2024-02-20T18:10:47.727955820-06:00 INFO lndk::onion_messenger - Consume messenger events received: messenger event: 02ad2e51564b763f65c934d67aa6528ad8238236691457c65b34fc49f843ec0662 incoming onion message.
2024-02-20T18:10:47.728183378-06:00 INFO lndk::onion_messenger - Peer: 02ad2e51564b763f65c934d67aa6528ad8238236691457c65b34fc49f843ec0662 hit rate limit, dropping incoming onion message
2024-02-20T18:10:47.729183480-06:00 INFO lndk::onion_messenger - Consume messenger events received: messenger event: 02ad2e51564b763f65c934d67aa6528ad8238236691457c65b34fc49f843ec0662 connected, onion message support: true.

Notice that we don't get the event that until after we receive the message.

[mauricepoirrier]

I'll give this a try.

Did you see this bug paying offers from which nodes (I bet Eclair)? Which nodes were in your regtest network connected to?