search

42wim / matterircd

Connect to your mattermost or slack using your IRC-client of choice.
MIT License
295 stars 60 forks source link

Is there a way to reduce the number of connections per second? #271

Closed Kerl13 closed 4 years ago

Kerl13 commented 4 years ago

I use matterircd to connect to a mattermost instance with a limit on the number of connections per seconds and I get banned very frequently because I'm spamming the API too much. Running matterircd behind mitmproxy indicates it peaks at about 35 connections/second but I suspect this rate is only an under-approximation as I don't get banned too often when using the proxy. I don't have this kind of problem with the official electron client (but it could be just because it is slow ^^).

So is there a way to reduce the number of requests per second made by matterircd? As a workaround it seems possible to do rate-limiting using external tools but they don't seem easy to use so I wanted to ask here first.

By the way besides this tiny problem, I'm very happy with matterircd, thanks for maintaining it :+1:

42wim commented 4 years ago

The number of connections per second should only be a lot during startup when it's fetching everything. I've just increased the amount of throttling on startup and there should be a lot of caching now too.

Can you retry with master?

Kerl13 commented 4 years ago

Sorry that I was a little slow to respond. I'm still being banned and yes this happens at startup (as of e48050493c667b75e1dae94c3ae419b7063f8d84)

Kerl13 commented 4 years ago

Looking at what you did the first time I got mattermost to not-ban me by increasing the amount of throttling a bit more:

diff --git a/mm-go-irckit/userbridge.go b/mm-go-irckit/userbridge.go
index 5000c56..d83c5b7 100644
--- a/mm-go-irckit/userbridge.go
+++ b/mm-go-irckit/userbridge.go
@@ -412,7 +412,7 @@ func (u *User) addUsersToChannels() {
        }

        srv := u.Srv
-       throttle := time.NewTicker(time.Millisecond * 200)
+       throttle := time.NewTicker(time.Millisecond * 400)

        logger.Debug("in addUsersToChannels()")
        // add all users, also who are not on channels

Is there any chance you might add an option to choose between say high and low network usage at startup? I get that it's a very specific use-case so I'm fine with a no. At least now I know where to patch for my use-case :slightly_smiling_face:

Cheers

42wim commented 4 years ago

@Kerl13 could you try the current master?

I've implemented rate-limiting on frequently used queries, so now it should play nice with mattermost ratelimits (and sleep the specified amount of time before retrying)

Kerl13 commented 4 years ago

I get banned shortly after logging in and then matterircd crashes with a segfault. The 503 error from mattermost is confusing, but I can't access the web interface unless I use another IP (and it works fine from the other IP).

$ ./matterircd
INFO[2020-08-30T18:38:45+02:00] Running version 0.20.0-dev                    module=matterircd
INFO[2020-08-30T18:38:45+02:00] WARNING: THIS IS A DEVELOPMENT VERSION. Things may break.  module=matterircd
INFO[2020-08-30T18:38:45+02:00] Listening on 127.0.0.1:6667                   module=matterircd
INFO[2020-08-30T18:38:57+02:00] New connection: 127.0.0.1:41848               module=matterircd
loggerlevel: info
INFO[0028] login as XXXXX (team: XXXXX) on XXXXX
[0028]  INFO matterclient: Found version 5.25.0.5.25.1.953947690adc02d5b1dd145e76806def.false
[0029]  INFO matterclient: found 57 users in team XXX
[0030]  INFO matterclient: found 331 users in team XXX
[0030]  INFO matterclient: found 70 users in team XXX
[0031]  INFO matterclient: found 434 users in team XXX
INFO[0031] login succeeded                              
[0031]  INFO matterclient: getting response: &model.WebSocketResponse{Status:"OK", SeqReply:1, Data:map[string]interface {}(nil), Error:(*model.AppError)(nil)}
AppErrorFromJson: model.utils.decode_json.app_error, body: <html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body bgcolor="white">
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.14.2</center>
</body>
</html>

AppErrorFromJson: model.utils.decode_json.app_error, body: <html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body bgcolor="white">
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.14.2</center>
</body>
</html>

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x865901]

goroutine 74 [running]:
github.com/42wim/matterircd/mm-go-irckit.(*User).addUserToChannelWorker(0xc0002140f0, 0xc00016d260, 0xc0000be640)
    /home/martin/git/matterircd/mm-go-irckit/userbridge.go:506 +0x201
created by github.com/42wim/matterircd/mm-go-irckit.(*User).addUsersToChannels
    /home/martin/git/matterircd/mm-go-irckit/userbridge.go:437 +0x332
42wim commented 4 years ago

@Kerl13 thanks for trying this, sorry to get you banned :/ This doesn't seem mattermost behaviour, as they only rate-limit, but something that nginx does?

You were talking about running with the mitmproxy, any way to use this again and see if any http request actually returns a 429 before you get banned ?

Normally mattermost should return a 409 and a X-RateLimit-Reset header.

(also fixed that panic in master)

Kerl13 commented 4 years ago

@Kerl13 thanks for trying this, sorry to get you banned :/

No worries, the ban-time is only a few minutes, I'm happy to keep trying things

This doesn't seem mattermost behaviour, as they only rate-limit, but something that nginx does?

Yes indeed. In the meantime I asked the sysadmins about the 503 error and they told me I was kicked out by nginx's limit_req.

You were talking about running with the mitmproxy, any way to use this again and see if any http request actually returns a 429 before you get banned ?

I just tried to run again with mitmproxy but it incurs too much slowdown and I don't hit the rate limit anymore.