search

coala / meta

A repository for non-code activities, such as engagement initiatives, and other meta issues
6 stars 5 forks source link

GitMate turned off 21 December 2017 #108

Closed jayvdb closed 6 years ago

jayvdb commented 6 years ago

The gitmate incident report is at https://gitlab.com/gitmate/open-source/gitmate-2/issues/275

To be consistent, times are in GMT+1

At 21 December 2017 07:00 @jayvdb turned off gitmate on all repositories.

The first incident event was on 20 December 18:59 @adtac (coala) reported the problematic activity on Gitter "GitMateIO/support", showing that merged PR coala/coala-eclipse#3 from 25-May-2016 was being labeled.

20 minutes later at 19:19 @nkprince007 raised https://gitlab.com/gitmate/open-source/gitmate-2/issues/271

6 hours later, at 21 December 02:19 @nkprince007 opened https://gitlab.com/gitmate/open-source/IGitt/merge_requests/191

5 hours later, at 21 December 2017 07:00 @jayvdb noticed the problem was re-occurring and shut down gitmate.

an hour later at 08:14 @sils merged https://gitlab.com/gitmate/open-source/IGitt/merge_requests/191 , hot fixes were deployed soon after, etc, see gitmate report mentioned above.

One aspect I am confident about there was that IGitt had no concept of 'merged' MergedRequests at the time of the incident, and only had states for 'open' & 'closed'. merged is a real status in GitLab, and doesnt exist in GitHub thus needs to be derived from the merged_at attribute. The merged status in GitLab was being completely ignored.

A coala related aspect is https://gitlab.com/gitmate/open-source/IGitt/merge_requests/163 , as it was a coala GCI task which added IGitt/Interfaces/Issue.IssueStates. That patch was about Issue states, which MergeRequests are a subclass of, so it updated some test cases for GitLab Merge Requests & GitHub Pull Requests, but only "open" and "closed" were being tested, so only those were updated. "merged" did not exist in IGitt, even though it was actually a real MR status in GitLab that had not been recognised.

Some code paths in IGitt or gitmate may have worked correctly if they checked only whether the MR state was 'open', and assumed all others were not open.

jayvdb commented 6 years ago

It seems reasonable to re-enable now. It would be wise to re-enable only on a few unimportant repos on GitHub & GitLab for 24 hrs as we dont know when any cron jobs are being run.

jayvdb commented 6 years ago

Just now I've enabled on

I'd like to at least wait 24 hrs and then go inspect those repos for oddities.

Moving around this UI is too painful for me, as the UI crawls with 3000+ repos.

If that is, maybe all that is required is someone to enable the repos and the old config will be re-activated.

nkprince007 commented 6 years ago

@jayvdb, a refresh of the UI is due soon. It will soon only show active repos and provides you with an option to add/remove them like all other CI. Instead of populating all of them.

jayvdb commented 6 years ago

@nkprince007 You've pinged the wrong person; JayVB instead of ... me.

jayvdb commented 6 years ago

Update, I've re-activated GitMate on a few coala GitHub repos (devops; frontend-landing, and docker). We've found that GitMate is not immediately removing the stale labels when there is activity. On cEPs, I fiddled with the GitMate settings a little, and it started working. We've encountered a similar problem with the docker repo, but havent pinned down the cause or the workaround yet.

sils commented 6 years ago

We've found that GitMate is not immediately removing the stale labels when there is activity.

Any example?

jayvdb commented 6 years ago

Discussion was on https://gitter.im/GitMateIO/support , with @nkprince007

sils commented 6 years ago

@jayvdb please note that GitMate doesn't count labelling or assignment changes as activity. Comments for sure and I think also if someone opens a pull request that mentions the issue.

As for GitMate-bot not appearing for the workshops repo: the repo wasn't configured, no team had access to the repo.

Makman2 commented 6 years ago

coala and coala-bears run gitmate again, however I deactivated as many non-crucial features as possible that affect labelling.

jayvdb commented 6 years ago

Most repos have been re-activated.

Closing as resolved.