"Your GitLab account was disconnected due to an invalid or revoked authorization token" #411

Open matthiasg opened 10 months ago

We are constantly getting the following message:

Your GitLab account was disconnected due to an invalid or revoked authorization token. Reconnect your account using the /git`lab connect command

calling gitlab connect works again for a day or two, and then it starts over for all users. gitlab v. 16.0.1 . The application is still registered in gitlab and running the connect commands works of course, for a while.

What should we debug ? At the moment we are updating to 16.5 of gitlab, but there seem to be no changes in the auth flow.

Hi @matthiasg, thanks for filing this issue. Did this start happening after an upgrade in the plugin or an upgrade of GitLab itself?

We're looking into this issue now. If we have any debug steps we'll report back here. Thanks

it happened before, i updated gitlab after updating mattermost and plugin to newest versions as of last week did not help.i have now updated gitlab to 16.5 and will reconnect today , so we will see, but as I said i do not see any changes to the auth flow in gitlab.

Some feedback. I updated gitlab as discussed, but I got disconnected again today.

This started happening for us roughly around after we updated to GitLab 16. I don't think we updated the Mattermost GitLab Plugin at that time.

Happens quite often for us as well (5 users)

I have been unable to sustain a gitlab connection in Mattermost for more than a couple of days. Most of the time it will disconnect exactly 2 hours after I originally establish the connection. If I try again immediately, it may last for one or a few days. The first date I was disconnected was September 15. It had worked fine for months before that. We are using Gitlab Enterprise. They keep it pretty up-to-date.

Hi @alekseyp @jayhartley, thank you for commenting here. May I ask:

What version of GitLab are you running? Do you know when any relevant upgrades to GitLab were made to your GitLab instance near September 15th?
Same question for the GItLab plugin for Mattermost.
Are you running in an HA environment?
Are you and/or your users using the Mattermost desktop app?
Do you and/or your users typically have multiple tabs of Mattermost open?
Would you be willing to run a debug build that adds verbose logging of all token state, with the actual access/refresh tokens sanitized?

What version of GitLab are you running? Do you know when any relevant upgrades to GitLab were made to your GitLab instance near September 15th?

Current: 16.6.2, but it started after one of the recent updates on Nov 16th.

Same question for the GItLab plugin for Mattermost.

GitLab(1.7.0)

Are you running in an HA environment?

No

Are you and/or your users using the Mattermost desktop app?

yes

Do you and/or your users typically have multiple tabs of Mattermost open?

I don't think so

Would you be willing to run a debug build that adds verbose logging of all token state, with the actual access/refresh tokens sanitized?

Yes

Thanks @alekseyp, we're actively looking into the cause of this and will keep you updated. We're releasing a new version of the plugin that may solve the issue, though since we haven't nailed down the exact cause between our code and GitLab's code, we aren't 100% sure it will solve the issue in all cases. Thanks for the details you've provided.

What version of GitLab are you running? Do you know when any relevant upgrades to GitLab were made to your GitLab instance near September 15th?

Gitlab enterprise 15.11.11-e. No updates around the time I started getting disconnected. The really strange thing is that, out of hundreds of users I seem to be the only one with this problem, according to my IT dept.

Same question for the GItLab plugin for Mattermost.

1.7.0

Are you running in an HA environment?

Not sure. I assume so.

Are you and/or your users using the Mattermost desktop app?

yes. Found the same problem when establishing the connection in browser.

Do you and/or your users typically have multiple tabs of Mattermost open?

I certainly don't - app only. Can't speak for others.

Would you be willing to run a debug build that adds verbose logging of all token state, with the actual access/refresh tokens sanitized?

I think so. Would need to check with IT.

@alekseyp @jayhartley There is a new version v1.8.0 that alleviates some issues related to simultaneous access to the tokens.

Can you please install this version to see if the problem is resolved?

Looks like new version is not on marketplace yet https://github.com/mattermost/mattermost-plugin-gitlab/issues/432

I can try to manually replace the files

I've manually installed 1.8.0 4 days ago and got logged out after couple of hours. 2nd logout just happened now, 4 days later.

--

Added:

2 more logouts today

I am reliant on our IT team to install the updates. I seem to be the only one experiencing this, or at least the only one who has brought it to IT, so it's not a high priority.

Interestingly, I consistently experience the same timing as reported by @alekseyp - initial disconnect almost exactly two hours after connecting. Then if I immediately re-connect, it works for about 4 days before kicking me out.

Still happening every day

The biggest pain point right now is that every time my account gets disconnected - the whole instance stops receiving webhooks/updates. Any idea if there is a way to fix that? Maybe create a separate user or some sort of direct API?

Hi @alekseyp, we're still actively working on this issue. Thank you for your patience.

The reason why the webhooks don't work in that case is that we're checking that the user that created the subscription has access to the project in the event, before showing the event post in the channel. We may change this to put this check only when creating the subscription, instead of also when events come into the system.

Any feedback?

Our team is considering migrating to another chat to get notifications in, all this due to the fact we have to reconnect 2-3 times a day right now

Guys, please fix the problem, it's very annoying.

We did not see this issue anymore since the last upgrade. Though i am not sure whether it was the plugin or the mattermost server

I just got a disconnect. Latest version of GitLab and the plugin.

Having to reconnect via web instead of app is weird as well, but that I can live with :D

Latest gitlab, latest plugin (1.9.0) - 6+ disconnects a day. We passed all previous stages of anger and just accepted this as is until we move to slack :D

Let it rain

The issue does not seem to be occurring for our instance anymore since a while now. We were affected before.

We're currently running

GitLab 16.11.1 Mattermost 9.7.3 Mattermost GitLab Plugin 1.9.0

I cannot tell what exactly made things better for us as I did not bother reconnecting my account to GitLab for a couple of months in the meantime.

@alekseyp Be sure to check if the other involved components are up to date on your side as this issue might very well be coming from one of those.

@alekseyp , have you subscribed to any repos after connecting? I haven't had any issues recently, but I also explicitly subscribed to a couple of repos.

@DaDummy everything is up to date.

@jayhartley yeah, a dozen of them.

GitLab 17.0.1 Mattermost 9.7.2 Mattermost GitLab Plugin 1.9.1

Disconnects multiple times a day

Hi @alekseyp, do you notice any pattern that causes the disconnect to occur? I presume it's happening during the processing of a webhook request. Can you please verify your webhook setup in GitLab? I'm asking because GitLab may be sending multiple requests in quick succession depending on how the webhooks are set up. I assume you have only one webhook (otherwise you would receive duplicate posts for subscriptions), so that's not the issue. Maybe the events configured on the webhook sometimes cause multiple requests to be sent in quick succession?

Is there any additional info from the server logs around this time that would help here?

@mickmister we have quite a few webhooks. Some are per project - others per group only (no duplicates, as far as I know, since like you said, we don't get double notifications).

I've tried to look into logs, but didn't see anything. If you want us to add some debug code, I don't mind at all, because it's very annoying for everyone.

As for the disconnect, the only visible thing I see is amount of disconnect messages we get when it happens. Sometimes is 1, mostly 3 and in other cases I've seen 10+ at the same time, so maybe that would mean something.

That could happened in the middle of the night, when no-one is working or while people using GitLab.

Another interesting observation, after we've decided to disable mattermost-gitlab plugin completely - our LA went down 10 times (8 cores, 16GB RAM). We are only 5 people using Mattermost, so it doesn't make any sense that plugin will create so much load. By comparing with past access logs, I've seen non stop request (100's per minute) to labels API that stopped as soon as plugin is disabled. Maybe it's somehow related to the disconnects, who knows..

mattermost / mattermost-plugin-gitlab

"Your GitLab account was disconnected due to an invalid or revoked authorization token" #411