Feature Request: use backoff time to throttle local clients

[adn77]

First, thank you for this fine piece of software! 😄

I am using it headless to allow Apple Mail clients access to a shared O365 Mailbox (before someone asks, setting it up as an Exchange mailbox leads to different sync states across the dozen clients using it).

As the mailbox size grows we're getting throttling messages more often: BAD Request is throttled. Suggested Backoff Time: 89117 milliseconds\r\n* BYE Connection closed. 14\r\n

I was hoping that the proxy might be able to slow down requests to local clients connecting as a throttled user. I don't really know any python or I would've tried myself.

Maybe this request doesn't make sense at all - it's under the assumption that the throttling increases, the more invalid requests are sent upstream.

[simonrob]

Thanks for this suggestion. It is an interesting idea, but quite a long way outside of the proxy's core aim (i.e., transparently handling OAuth).

The reason you're receiving this throttling message is nothing to do with the proxy – your client(s) are opening too many connections, or hitting some other activity limit within O365. As far as I'm aware, the limits aren't actually documented anywhere, but you may be able to request an increase from Microsoft for your account.

In terms of the proxy, you might be interested in the plugins branch, which could allow detection and some sort of response to this behaviour. Could you explain a bit more about what you're hoping to achieve, though?

Given that this comes straight from O365, what would be the difference between the client receiving this response and it receiving some other error message from the proxy? Or were you hoping to, say, proactively deny additional connections by the same client above some limit? I've no idea what Apple Mail does in this scenario, but if it is not respecting the suggested backoff time, then I can see how locally rejecting its connections (instead of letting them through to O365) would be useful.

A bit more detail about your idea would be interesting either way.

[adn77]

I totally agree, this has nothing to do with the proxy - I was just trying to put a bandaid on Microsoft's mis-behavings.

My idea is that throttling increases the more the backoff time is disrespected. Since Apple Mail ignores the backoff my hope was to delay clients by means of the proxy.

The plugin branch looks promising. We could match for the Backoff Time: ([0-9]+) milliseconds and delay the next response to the connected client by that amount. In order to prevent any timeouts by the client, we could sent an * OK still trying... every 10 seconds until the Backoff Time is over. Ideally we could pickup where the client left off or just force the connection close. I don't know what that sleep would do to the multi-threaded nature of your program though...

I am in no way qualified to do IMAP server programming - I briefly examined the IMAP server responses of RFC3501 (https://datatracker.ietf.org/doc/html/rfc3501#section-7.1.1) 🙂

[simonrob]

Thanks for the extra detail. It would be fairly straightforward to create a plugin that does a basic version of this – nothing quite as complex as picking up where the connection left off, but either a) delaying the actual backoff response to the client so that its next communication happens after the backoff time; or, b) locally rejecting future connections by the same client when throttling is detected until the backoff time expires.

For option a, there's a high chance that the client would disconnect or time out before the backoff time expires. There may be ways to alleviate this like you say, and sleeping would be fine on the proxy side because each connection is isolated. But the other issue is that most clients make multiple connections, and this option would not have any impact on those.

For option b, the main challenge would be detecting when the same client connects. Ideally, problematic clients would send a unique greeting that the proxy could intercept and use as a reference. To help with this, could you provide a more complete proxy log that includes the connection setup and also the communication around the throttling messages?

[simonrob]

Just following up about this – if you can provide a full log then this may be possible; if not then I'll close the issue as I don't have enough information to take a look. Let me know?

[adn77]

What can I say... Microsoft changed something and I don't have any log output to show. That's good and bad news 😄

Sorry to bother - if things become unbearable (again) at some point I will remember this issue and re-open it.

Thanks again for all your work!