Closed manoelpqueiroz closed 3 years ago
There have been quite a few issues with Gitlab.com today: https://status.gitlab.com/pages/history/5b36dc6502d06804c08349f7, you may want to retry ;-)
Seems like I'm gifted with dealing with awesome software on terrible days. :sweat_smile:
It looks like the instability still goes on for me, unfortunately, but this time is a 405 error.
ERROR:bugwarrior.services:Worker for [my_gitlab] failed: Non-200 status code 405; 'https://gitlab.com/api/v4/projects?simple=True&page=501&per_page=100'; '{"error":"Offset pagination has a maximum allowed offset of 50000 for requests that return objects of type Project. Remaining records can be retrieved using keyset pagination."}'
Traceback (most recent call last):
File "/home/myuser/.local/lib/python3.9/site-packages/bugwarrior/services/__init__.py", line 504, in _aggregate_issues
for issue in service.issues():
File "/home/myuser/.local/lib/python3.9/site-packages/bugwarrior/services/gitlab.py", line 484, in issues
all_repos = self._fetch_paged(tmpl + '?' + urlencode(querystring))
File "/home/myuser/.local/lib/python3.9/site-packages/bugwarrior/services/gitlab.py", line 377, in _fetch_paged
items = self._fetch(tmpl, params=params)
File "/home/myuser/.local/lib/python3.9/site-packages/bugwarrior/services/gitlab.py", line 366, in _fetch
return self.json_response(response)
File "/home/myuser/.local/lib/python3.9/site-packages/bugwarrior/services/__init__.py", line 481, in json_response
raise IOError(
OSError: Non-200 status code 405; 'https://gitlab.com/api/v4/projects?simple=True&page=501&per_page=100'; '{"error":"Offset pagination has a maximum allowed offset of 50000 for requests that return objects of type Project. Remaining records can be retrieved using keyset pagination."}'
INFO:bugwarrior.services:Done with [my_gitlab] in 1128.743913s
INFO:bugwarrior.services:Terminating workers
ERROR:bugwarrior.command:Aborted (critical error in target 'my_gitlab')
Traceback (most recent call last):
File "/home/myuser/.local/lib/python3.9/site-packages/bugwarrior/command.py", line 73, in pull
synchronize(issue_generator, config, main_section, dry_run)
File "/home/myuser/.local/lib/python3.9/site-packages/bugwarrior/db.py", line 357, in synchronize
for issue in issue_generator:
File "/home/myuser/.local/lib/python3.9/site-packages/bugwarrior/services/__init__.py", line 572, in aggregate_issues
raise RuntimeError(
RuntimeError: critical error in target 'my_gitlab'
Guys, is there something I'm missing here?
The 500 error still persists though.
Any advice to better log what might be happening? I'd like to be able to provide you with better information than just say it's not working, I never feel right when I just relay superficial information for developers (I know that's the worst thing that can come from non-developing users :disappointed:).
My best guess is that when you exceeded the pagination limits and received the 405 (perhaps multiple times) that caused one of gitlab's server's designed to defend against abuse to flag you and return a 500 when you go on future paging sprees. We could expand bugwarrior's pagination capability by switching to keyset pagination but I'm doubtful anybody actually needs that. The real question is: why is bugwarrior requesting a list of over 50,000 projects (page 501 with 100 per page) when you only have 5?
I believe it's because you aren't applying any filter to the projects so bugwarrior is trying to go through all the public projects on the server. This is insane for gitlab.com but probably fine for most private gitlab instances. We should probably add a sanity check to prevent people from encountering this confusing error message though (#813).
However, the filter I believe you want is undocumented because our documentation is like 3 years out of date (#812). At any rate, I think you want to add to your configuration:
gitlab.owned = True
Thanks for your persistence and let us know if that solves your problem.
Re keyset pagination: they're only rolling it out on an API-per-API basis, so you need to make sure the API that you request actually supports it. I've tried it before, and it just silently falls back to standard pagination if it's not yet activated. Very annoying, although transitive.
Maybe it's an idea to fire a warning if you don't have any sane filters? I went through this problem too, a few years ago, and I imagine a lot of BugWarrior+Gitlab users go through this.
That was it, @ryneeverett! Bummed that the problem here was located between the chair and the keyboard :sweat_smile:, but glad that this could also help you guys improve Bugwarrior.
Hope to also contribute in the near future and thank you for you patience as well.
So I set up the rc file inside
.config/bugwarrior/bugwarriorrc
to work with a GitLab token enabled for read_user, read_api, read_repository and read_registry:But calling
bugwarrior-pull
never went past the line stating it was working on Gitlab: no errors or additional messages after 3 runs and aborting after a minute or so. Given that I only have 5 repositories on my user and only one of them has issues (3, in its case), my thoughts it can't be latency in the responses from the GitLab server.I ended up forcefully interrupting, but I never found out how to output more messages to try and capture the exact cause of this (the
--debug
flag also never showed more than these four lines).I tried to test out if it wasn't a connection problem, but Bugwarrior responded accordingly when I revoked my access token in the middle of the run:
I eventually left it running and at last it returned an error code 500 from the server:
Am I missing something here? I have successfully managed to get some response using
requests
on the projects endpoint (same token), not sure if this has to do with the endpoint or if Bugwarrior tries to access v3 instead of v4 in Gitlab.If there is anything I can contribute with coding I'll be glad to help.