Open micheelengronne opened 4 years ago
@micheelengronne This is probably due to the fact that you're running GitLab Core (CE) which have no approvers and therefor get's a 404 on the GitLab API.
Actually Marge-bot is compatible with GitLab Core (CE).
I found this bug in GitLab Core:
$ curl -H "PRIVATE-TOKEN: xxx" https://gitlab.yourdomain.com/api/v4/version
{"version":"12.6.0-ee","revision":""}
The -ee
causes the problem because there is an EE check in the approvals logic: https://github.com/smarkets/marge-bot/blob/0.9.1/marge/approvals.py#L17
This 12.6.0-ee
value is hardcoded in the installation /VERSION.
I've managed to change this on my installation to 12.6.0
and now this error is gone. Now there's another issue where the PUT /api/v4/projects/:id/merge_requests/:iid/merge does not result in a merge:
...
INFO Giving 10 more secs for !90 to be merged...
INFO Giving 10 more secs for !90 to be merged...
INFO Giving 10 more secs for !90 to be merged...
INFO Giving 10 more secs for !90 to be merged...
...
So is it a GitLab bug? I mean we have CE version with -ee
suffix reported now on 12.6 and Marge is broken. It would be nice to have some way to disable approvers functionality in config.
I have a same problem 😢
I found this bug in GitLab Core:
$ curl -H "PRIVATE-TOKEN: xxx" https://gitlab.yourdomain.com/api/v4/version {"version":"12.6.0-ee","revision":""}
I found a related issue at the GitLab community. But this fix looks like to take some time... https://gitlab.com/gitlab-com/gl-infra/delivery/issues/582
This
12.6.0-ee
value is hardcoded in the installation /VERSION.I've managed to change this on my installation to
12.6.0
and now this error is gone. Now there's another issue where the PUT /api/v4/projects/:id/merge_requests/:iid/merge does not result in a merge:... INFO Giving 10 more secs for !90 to be merged... INFO Giving 10 more secs for !90 to be merged... INFO Giving 10 more secs for !90 to be merged... INFO Giving 10 more secs for !90 to be merged... ...
Yep, there is another problem: PUT request to /merge
isn't changing merge_when_pipeline_succeeds
to true. It's still false after 200 OK response, see logs:
2019-12-24 09:08:39,023 INFO Ensuring MR !1881 is mergeable
2019-12-24 09:08:39,024 DEBUG Ensuring MR MergeRequest(<marge.gitlab.Api object at 0x7f9f127c6588>, {'id': 3035, 'iid': 1881, 'project_id': 7, 'title': 'SP-6829: \u041f\u0440\u0438\u0432\u043e\u0434\u0438\u043c \u0432 \u043f\u043e\u0440\u044f\u0434\u043e\u043a \u043e\u0442\u043f\u0440\u0430\u0432\u043a\u0443 \u0441\u043e\u043e\u0431\u0449\u0435\u043d\u0438\u0439 \u043f\u043e \u0431\u044b\u0441\u0442\u0440\u044b\u043c \u043f\u043e\u0434\u043f\u0438\u0441\u043a\u0430\u043c', 'description': '', 'state': 'opened', 'created_at': '2019-12-24T13:37:10.983+10:00', 'updated_at': '2019-12-24T19:02:05.011+10:00', 'merged_by': None, 'merged_at': None, 'closed_by': None, 'closed_at': None, 'target_branch': 'development', 'source_branch': 'SP-6829', 'user_notes_count': 3, 'upvotes': 0, 'downvotes': 0, 'assignee': {'id': 10, 'name': '\xa0Marge Bot', 'username': 'marge-bot', 'state': 'active', 'avatar_url': 'https://redacted/uploads/-/system/user/avatar/10/avatar.png', 'web_url': 'https://redacted/marge-bot'}, 'author': {'id': 6, 'name': 'redacted redacted', 'username': 'redacted', 'state': 'active', 'avatar_url': 'https://redacted/uploads/-/system/user/avatar/6/avatar.png', 'web_url': 'https://redacted/redacted'}, 'assignees': [{'id': 10, 'name': '\xa0Marge Bot', 'username': 'marge-bot', 'state': 'active', 'avatar_url': 'https://redacted/uploads/-/system/user/avatar/10/avatar.png', 'web_url': 'https://redacted/marge-bot'}], 'source_project_id': 7, 'target_project_id': 7, 'labels': ['approved'], 'work_in_progress': False, 'milestone': None, 'merge_when_pipeline_succeeds': False, 'merge_status': 'can_be_merged', 'sha': '46aa6e92ba98c078afcd42dac97971c4c700c241', 'merge_commit_sha': None, 'squash_commit_sha': None, 'discussion_locked': None, 'should_remove_source_branch': None, 'force_remove_source_branch': True, 'reference': '!1881', 'web_url': 'https://redacted/redacted/redacted.ru/merge_requests/1881', 'time_stats': {'time_estimate': 0, 'total_time_spent': 0, 'human_time_estimate': None, 'human_total_time_spent': None}, 'squash': True, 'task_completion_status': {'count': 0, 'completed_count': 0}, 'has_conflicts': False, 'blocking_discussions_resolved': True, 'subscribed': True, 'changes_count': '5', 'latest_build_started_at': '2019-12-24T19:01:00.412+10:00', 'latest_build_finished_at': '2019-12-24T19:08:32.239+10:00', 'first_deployed_to_production_at': None, 'pipeline': {'id': 21831, 'sha': '46aa6e92ba98c078afcd42dac97971c4c700c241', 'ref': 'refs/merge-requests/1881/head', 'status': 'success', 'created_at': '2019-12-24T19:00:57.361+10:00', 'updated_at': '2019-12-24T19:08:32.249+10:00', 'web_url': 'https://redacted/redacted/redacted.ru/pipelines/21831'}, 'head_pipeline': {'id': 21831, 'sha': '46aa6e92ba98c078afcd42dac97971c4c700c241', 'ref': 'refs/merge-requests/1881/head', 'status': 'success', 'created_at': '2019-12-24T19:00:57.361+10:00', 'updated_at': '2019-12-24T19:08:32.249+10:00', 'web_url': 'https://redacted/redacted/redacted.ru/pipelines/21831', 'before_sha': '0000000000000000000000000000000000000000', 'tag': False, 'yaml_errors': None, 'user': {'id': 10, 'name': '\xa0Marge Bot', 'username': 'marge-bot', 'state': 'active', 'avatar_url': 'https://redacted/uploads/-/system/user/avatar/10/avatar.png', 'web_url': 'https://redacted/marge-bot'}, 'started_at': '2019-12-24T19:01:00.412+10:00', 'finished_at': '2019-12-24T19:08:32.239+10:00', 'committed_at': None, 'duration': 449, 'coverage': None, 'detailed_status': {'icon': 'status_success', 'text': 'passed', 'label': 'passed', 'group': 'success', 'tooltip': 'passed', 'has_details': True, 'details_path': '/redacted/redacted.ru/pipelines/21831', 'illustration': None, 'favicon': '/assets/ci_favicons/favicon_status_success-8451333011eee8ce9f2ab25dc487fe24a8758c694827a582f17f42b0a90446a2.png'}}, 'diff_refs': {'base_sha': 'ab9fa6a78963309c51c82188145efd2dd0850555', 'head_sha': '46aa6e92ba98c078afcd42dac97971c4c700c241', 'start_sha': 'ab9fa6a78963309c51c82188145efd2dd0850555'}, 'merge_error': None, 'user': {'can_merge': True}}) is mergeable
2019-12-24 09:08:39,024 DEBUG REQUEST: GET https://redacted/api/v4/version {'PRIVATE-TOKEN': 'aFo7R1siuA52-j2dpbqp'} {'params': {}}
2019-12-24 09:08:39,026 DEBUG Starting new HTTPS connection (1): redacted:443
2019-12-24 09:08:39,063 DEBUG https://redacted:443 "GET /api/v4/version HTTP/1.1" 200 45
2019-12-24 09:08:39,066 DEBUG RESPONSE CODE: 200
2019-12-24 09:08:39,067 DEBUG RESPONSE BODY: b'{"version":"12.6.0","revision":"fc376e40baf"}'
2019-12-24 09:08:39,068 DEBUG REQUEST: PUT https://redacted/api/v4/projects/7/merge_requests/1881/merge {'PRIVATE-TOKEN': 'aFo7R1siuA52-j2dpbqp'} {'json': {'should_remove_source_branch': True, 'merge_when_pipeline_succeeds': True, 'sha': '46aa6e92ba98c078afcd42dac97971c4c700c241'}}
2019-12-24 09:08:39,070 DEBUG Starting new HTTPS connection (1): redacted:443
2019-12-24 09:08:39,273 DEBUG https://redacted:443 "PUT /api/v4/projects/7/merge_requests/1881/merge HTTP/1.1" 200 3543
2019-12-24 09:08:39,276 DEBUG RESPONSE CODE: 200
2019-12-24 09:08:39,276 DEBUG RESPONSE BODY: b'{"id":3035,"iid":1881,"project_id":7,"title":"SP-6829: \xd0\x9f\xd1\x80\xd0\xb8\xd0\xb2\xd0\xbe\xd0\xb4\xd0\xb8\xd0\xbc \xd0\xb2 \xd0\xbf\xd0\xbe\xd1\x80\xd1\x8f\xd0\xb4\xd0\xbe\xd0\xba \xd0\xbe\xd1\x82\xd0\xbf\xd1\x80\xd0\xb0\xd0\xb2\xd0\xba\xd1\x83 \xd1\x81\xd0\xbe\xd0\xbe\xd0\xb1\xd1\x89\xd0\xb5\xd0\xbd\xd0\xb8\xd0\xb9 \xd0\xbf\xd0\xbe \xd0\xb1\xd1\x8b\xd1\x81\xd1\x82\xd1\x80\xd1\x8b\xd0\xbc \xd0\xbf\xd0\xbe\xd0\xb4\xd0\xbf\xd0\xb8\xd1\x81\xd0\xba\xd0\xb0\xd0\xbc","description":"","state":"opened","created_at":"2019-12-24T13:37:10.983+10:00","updated_at":"2019-12-24T19:02:05.011+10:00","merged_by":null,"merged_at":null,"closed_by":null,"closed_at":null,"target_branch":"development","source_branch":"SP-6829","user_notes_count":3,"upvotes":0,"downvotes":0,"assignee":{"id":10,"name":"\xc2\xa0Marge Bot","username":"marge-bot","state":"active","avatar_url":"https://redacted/uploads/-/system/user/avatar/10/avatar.png","web_url":"https://redacted/marge-bot"},"author":{"id":6,"name":"redacted redacted","username":"redacted","state":"active","avatar_url":"https://redacted/uploads/-/system/user/avatar/6/avatar.png","web_url":"https://redacted/redacted"},"assignees":[{"id":10,"name":"\xc2\xa0Marge Bot","username":"marge-bot","state":"active","avatar_url":"https://redacted/uploads/-/system/user/avatar/10/avatar.png","web_url":"https://redacted/marge-bot"}],"source_project_id":7,"target_project_id":7,"labels":["approved"],"work_in_progress":false,"milestone":null,"merge_when_pipeline_succeeds":false,"merge_status":"can_be_merged","sha":"46aa6e92ba98c078afcd42dac97971c4c700c241","merge_commit_sha":null,"squash_commit_sha":null,"discussion_locked":null,"should_remove_source_branch":null,"force_remove_source_branch":true,"reference":"!1881","web_url":"https://redacted/redacted/redacted.ru/merge_requests/1881","time_stats":{"time_estimate":0,"total_time_spent":0,"human_time_estimate":null,"human_total_time_spent":null},"squash":true,"task_completion_status":{"count":0,"completed_count":0},"has_conflicts":false,"blocking_discussions_resolved":true,"subscribed":true,"changes_count":"5","latest_build_started_at":"2019-12-24T19:01:00.412+10:00","latest_build_finished_at":"2019-12-24T19:08:32.239+10:00","first_deployed_to_production_at":null,"pipeline":{"id":21831,"sha":"46aa6e92ba98c078afcd42dac97971c4c700c241","ref":"refs/merge-requests/1881/head","status":"success","created_at":"2019-12-24T19:00:57.361+10:00","updated_at":"2019-12-24T19:08:32.249+10:00","web_url":"https://redacted/redacted/redacted.ru/pipelines/21831"},"head_pipeline":{"id":21831,"sha":"46aa6e92ba98c078afcd42dac97971c4c700c241","ref":"refs/merge-requests/1881/head","status":"success","created_at":"2019-12-24T19:00:57.361+10:00","updated_at":"2019-12-24T19:08:32.249+10:00","web_url":"https://redacted/redacted/redacted.ru/pipelines/21831","before_sha":"0000000000000000000000000000000000000000","tag":false,"yaml_errors":null,"user":{"id":10,"name":"\xc2\xa0Marge Bot","username":"marge-bot","state":"active","avatar_url":"https://redacted/uploads/-/system/user/avatar/10/avatar.png","web_url":"https://redacted/marge-bot"},"started_at":"2019-12-24T19:01:00.412+10:00","finished_at":"2019-12-24T19:08:32.239+10:00","committed_at":null,"duration":449,"coverage":null,"detailed_status":{"icon":"status_success","text":"passed","label":"passed","group":"success","tooltip":"passed","has_details":true,"details_path":"/redacted/redacted.ru/pipelines/21831","illustration":null,"favicon":"/assets/ci_favicons/favicon_status_success-8451333011eee8ce9f2ab25dc487fe24a8758c694827a582f17f42b0a90446a2.png"}},"diff_refs":{"base_sha":"ab9fa6a78963309c51c82188145efd2dd0850555","head_sha":"46aa6e92ba98c078afcd42dac97971c4c700c241","start_sha":"ab9fa6a78963309c51c82188145efd2dd0850555"},"merge_error":null,"user":{"can_merge":true}}'
So that's the problem. I don't know if it's a GitLab problem/bug or something wrong on the bot's side.
I tested /merge
PUT request using curl and it worked as it should, merge request merged… Switching back to 12.5 for now, probably. Looking forward to a fix.
curl -X PUT --header "PRIVATE-TOKEN: aFo7R1siuA52-j2dpbqp" -d "{'json': {'should_remove_source_branch': True, 'merge_when_pipeline_succeeds': True, 'sha': '46aa6e92ba98c078afcd42dac97971c4c700c241'}}" https://gitlab.redacted.ru/api/v4/projects/7/merge_requests/1881/merge
{"id":3035,"iid":1881,"project_id":7,"title":"SP-6829: Приводим в порядок отправку сообщений по быстрым подпискам","description":"","state":"merged","created_at":"2019-12-24T13:37:10.983+10:00","updated_at":"2019-12-24T19:27:57.956+10:00","merged_by":{"id":10,"name":" Marge Bot","username":"marge-bot","state":"active","avatar_url":"https://gitlab.redacted.ru/uploads/-/system/user/avatar/10/avatar.png","web_url":"https://gitlab.redacted.ru/marge-bot"},"merged_at":"2019-12-24T19:27:58.022+10:00","closed_by":null,"closed_at":null,"target_branch":"development","source_branch":"SP-6829","user_notes_count":4,"upvotes":0,"downvotes":0,"assignee":{"id":6,"name":"redacted redacted","username":"redacted","state":"active","avatar_url":"https://gitlab.redacted.ru/uploads/-/system/user/avatar/6/avatar.png","web_url":"https://gitlab.redacted.ru/redacted"},"author":{"id":6,"name":"redacted redacted","username":"redacted","state":"active","avatar_url":"https://gitlab.redacted.ru/uploads/-/system/user/avatar/6/avatar.png","web_url":"https://gitlab.redacted.ru/redacted"},"assignees":[{"id":6,"name":"redacted redacted","username":"redacted","state":"active","avatar_url":"https://gitlab.redacted.ru/uploads/-/system/user/avatar/6/avatar.png","web_url":"https://gitlab.redacted.ru/redacted"}],"source_project_id":7,"target_project_id":7,"labels":["approved"],"work_in_progress":false,"milestone":null,"merge_when_pipeline_succeeds":false,"merge_status":"can_be_merged","sha":"46aa6e92ba98c078afcd42dac97971c4c700c241","merge_commit_sha":"4c2d7731c1ca5c40dec91ce3364d4103f124aa4b","squash_commit_sha":null,"discussion_locked":null,"should_remove_source_branch":null,"force_remove_source_branch":true,"reference":"!1881","web_url":"https://gitlab.redacted.ru/redacted/redacted.ru/merge_requests/1881","time_stats":{"time_estimate":0,"total_time_spent":0,"human_time_estimate":null,"human_total_time_spent":null},"squash":true,"task_completion_status":{"count":0,"completed_count":0},"has_conflicts":false,"blocking_discussions_resolved":true,"subscribed":true,"changes_count":"5","latest_build_started_at":"2019-12-24T19:01:00.412+10:00","latest_build_finished_at":"2019-12-24T19:08:32.239+10:00","first_deployed_to_production_at":null,"pipeline":{"id":21831,"sha":"46aa6e92ba98c078afcd42dac97971c4c700c241","ref":"refs/merge-requests/1881/head","status":"success","created_at":"2019-12-24T19:00:57.361+10:00","updated_at":"2019-12-24T19:08:32.249+10:00","web_url":"https://gitlab.redacted.ru/redacted/redacted.ru/pipelines/21831"},"head_pipeline":{"id":21831,"sha":"46aa6e92ba98c078afcd42dac97971c4c700c241","ref":"refs/merge-requests/1881/head","status":"success","created_at":"2019-12-24T19:00:57.361+10:00","updated_at":"2019-12-24T19:08:32.249+10:00","web_url":"https://gitlab.redacted.ru/redacted/redacted.ru/pipelines/21831","before_sha":"0000000000000000000000000000000000000000","tag":false,"yaml_errors":null,"user":{"id":10,"name":" Marge Bot","username":"marge-bot","state":"active","avatar_url":"https://gitlab.redacted.ru/uploads/-/system/user/avatar/10/avatar.png","web_url":"https://gitlab.redacted.ru/marge-bot"},"started_at":"2019-12-24T19:01:00.412+10:00","finished_at":"2019-12-24T19:08:32.239+10:00","committed_at":null,"duration":449,"coverage":null,"detailed_status":{"icon":"status_success","text":"passed","label":"passed","group":"success","tooltip":"passed","has_details":true,"details_path":"/redacted/redacted.ru/pipelines/21831","illustration":null,"favicon":"/assets/ci_favicons/favicon_status_success-8451333011eee8ce9f2ab25dc487fe24a8758c694827a582f17f42b0a90446a2.png"}},"diff_refs":{"base_sha":"ab9fa6a78963309c51c82188145efd2dd0850555","head_sha":"46aa6e92ba98c078afcd42dac97971c4c700c241","start_sha":"ab9fa6a78963309c51c82188145efd2dd0850555"},"merge_error":null,"user":{"can_merge":true}}
ps. I'm not sure if curl request is right, maybe merge_when_pipeline_succeeds
wasn't applied, as it's optional.
@b1rdex Good catch. I also think that's a GitLab bug.
There is already an issue for it: https://gitlab.com/gitlab-org/gitlab/issues/121620
I'm also experiencing this. I think the API does work if the merge_when_pipeline_succeeds
parameter is absent from the request body. I'm considering doing a temporary fork which removes this parameter until Gitlab can get their issues worked out.
I'm curious why does Marge set this parameter? I see from the logs that Marge has already waited for CI to pass so by this point we should be at least somewhat certain that the pipeline should be finished. Is it just insurance in case there are race conditions?
@whoward when you trigger this bug, does Marge post I couldn't merge this branch: It is taking too long to see the request marked as merged!
on your MR or something else? I'm just asking because I'm not sure whether the behaviour I'm seeing is this bug or a separate issue!
@snim2 yes this is the issue we see, it could be possible it's caused by something else for you but in our case it was caused by the merge_when_pipeline_succeeds
param being passed.
The problem is that Marge hits the /merge
endpoint with with this param which due to the gitlab bug doesn't actually cause the merge request to be merged. Because of this Marge sits spinning waiting for the merge request to appear merged and eventually times out - triggering this message you saw.
I did a private fork of marge bot and removed this line: https://github.com/smarkets/marge-bot/blob/6f0a8f1ea138c83a4c02d995dc4e8e161d0f5cb4/marge/merge_request.py#L166 as well as the appropriate lines from the tests and our fork has been working in our development environment since Friday.
Keep in mind we tend to run close to the latest version of gitlab at all times and we tend to have a single common approval / merge process...so what works for me may not work for you.
@whoward many thanks for that. I suspect we do have the same problem, at least there's nothing in the debug logs to suggest that anything else is going on.
For me, it posts I'm broken on the inside, please somebody fix me... 😢
@micheelengronne you should enable debug mode and check logs to understand what actually is happening in your case.
@b1rdex I started the issue.
@b1rdex I started the issue.
Aw, sorry, didn’t notice that :D
For anyone in need of a quicker fix, gitlab!22777 fixes marge-bot when cherry-picked into GitLab 12.6.3 CE.
Fixed in the 12.7 version of GitLab.
@b1rdex For clarification: the merge_when_pipeline_succeeds
parameter bug is fixed, but the issue that caused the 404 in the GitLab API is still there.
Users of GitLab EE could upgrade to 12.7.
The original 404 issue remains for GitLab CE users. See gitlab-com/gl-infra/delivery#582.
Hm. We’re on the CE edition and the bug is gone completely.
Отправлено с iPhone
25 янв. 2020 г., в 23:26, Maikel notifications@github.com написал(а):
@b1rdex For clarification: the merge_when_pipeline_succeeds parameter bug is fixed, but the issue that caused the 404 in the GitLab API is still there.
Users of GitLab EE could upgrade to 12.7. The original 404 issue remains for GitLab CE users. See gitlab-com/gl-infra/delivery#582.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.
@b1rdex You're right. I assumed since gitlab-com/gl-infra/delivery#582 isn't closed.
Seems this was fixed in v12.6.1
$ docker run -it --rm registry.gitlab.com/gitlab-org/build/cng/gitlab-unicorn-ce:v12.6.0 cat /srv/gitlab/VERSION
12.6.0-ee
$ docker run -it --rm registry.gitlab.com/gitlab-org/build/cng/gitlab-unicorn-ce:v12.6.1 cat /srv/gitlab/VERSION
12.6.1
As soon as I updated my gitlab to 12.5, marge-bot broke with:
and this message on the MR comments:
I'm broken on the inside, please somebody fix me... 😢