Avoid 404 errors for groups we can see but not access

[fmartingr]

Fixed an uncaught exception that was caused when the tree is generating and we see a group we can't access. We tried to retrieve it's info but it raised an exception and that made the script fail completely (even if the group wasn't part of the -i flag.)

I've catched the exception and logged the error for the user to notice.

[ezbz]

Can we relate this to an issue? I'd like to try and reproduce before I merge

[fmartingr]

Hey @ezbz, this happened at my company because I could see some repos but not access to them (it 404's on the browser but I could see them on the listings for the group I belong to). Sadly I don't know the specifics as I'm not maintaining that part and it was "fixed" overnight.

[devstek]

Looks like my problem #74

[fmartingr]

Looks like my problem #74

Yeah that's look totally what happened to me.

[TheKangaroo]

Is this going to be mereged? I ran in the same issue :(

[jgroffen]

I need this as well - GitLab is showing top level groups to users that don't have priv to see them, but then returning a 404. Privs are correct and the user shouldn't see the group at all so this looks like a GitLab bug, and this PR would solve the issue.

[odin568]

Same issue, can this be merged and released? Would really appreciate @ezbz ❤️

My GitLab version is GitLab Enterprise Edition 13.12.9-ee - anyone knows if there is a bug ticket open on GitLab side? Did not find any so far

[odin568]

Ok, my issue is different. I fail on listing GitLab subgroups already: def get_subgroups(self, group, parent): fails with GitlabListError: 404

Therefor I adapted calling method:

    def load_gitlab_tree(self):
        groups = self.gitlab.groups.list(as_list=False, archived=self.archived)
        self.progress.init_progress(len(groups))
        for group in groups:
            if group.parent_id is None:
                group_id = group.name if self.naming == FolderNaming.NAME else group.path
                node = self.make_node(group_id, self.root, url=group.web_url)
                self.progress.show_progress(node.name, 'group')
                try:
                    self.get_subgroups(group, node)
                except GitlabListError as error:
                    if error.response_code == 404:
                        log.error(f"Error listing subgroup {group.name} (ID: {group.id}): {error}. Check your permissions as you may not have access to it.")
                        continue
                    else:
                        raise error
                self.get_projects(group, node)
        elapsed = self.progress.finish_progress()
        log.debug("Loading projects tree from gitlab took [%s]", elapsed)

Don't know if it is the best solution :(

@fmartingr May I ask you to adapt in your PR too? I'm a python noobie :)

[fmartingr]

@fmartingr May I ask you to adapt in your PR too? I'm a python noobie :)

Hey, I've just updated the PR with a handler for the list error too, can you test it on your use case? It works on mine.

[fmartingr]

While the other change is also fine IMHO, the error occurs alredy in this line, which makes it more tricky. Because just catching in this line and returning will fail then in the get_projects call afterwards. That's why I wrapped it in the calling method as a pragmatic solution...

(Line 109/110, The subgroups = group.subgroups.list(as_list=False, archived=self.archived) line

Hey sorry! I just did a quick glance of your comment and assumed was on the same method I was editing here. Sorry about that. Let me check this again tonight or tomorrow once I have more time to review it.

[fmartingr]

While the other change is also fine IMHO, the error occurs alredy in this line, which makes it more tricky. Because just catching in this line and returning will fail then in the get_projects call afterwards. That's why I wrapped it in the calling method as a pragmatic solution...

(Line 109/110, The subgroups = group.subgroups.list(as_list=False, archived=self.archived) line

Implemented your suggestion, can you check it out please?

[devstek]

I've tested it as well, i recieve another exception in get_projects()

gitlabber -t *** -p -u https://git***                
* loading tree: 1.1%|▉                                                                               | 1/89, group=***2021-12-21 12:53:17,753 - gitlabber.gitlab_tree - ERROR - Error listing group *** (ID: 12414): 404: 404 Group Not Found. Check your permissions as you may not have access to it.
Traceback (most recent call last):
  File "/Users/***/Library/Python/3.7/lib/python/site-packages/gitlab/exceptions.py", line 279, in wrapped_f
    return f(*args, **kwargs)
  File "/Users/***/Library/Python/3.7/lib/python/site-packages/gitlab/mixins.py", line 141, in list
    obj = self.gitlab.http_list(path, **data)
  File "/Users/***/Library/Python/3.7/lib/python/site-packages/gitlab/__init__.py", line 656, in http_list
    return GitlabList(self, url, query_data, **kwargs)
  File "/Users/***/Library/Python/3.7/lib/python/site-packages/gitlab/__init__.py", line 784, in __init__
    self._query(url, query_data, **self._kwargs)
  File "/Users/***/Library/Python/3.7/lib/python/site-packages/gitlab/__init__.py", line 789, in _query
    result = self._gl.http_request("get", url, query_data=query_data, **kwargs)
  File "/Users/***/Library/Python/3.7/lib/python/site-packages/gitlab/__init__.py", line 575, in http_request
    response_body=result.content,
gitlab.exceptions.GitlabHttpError: 404: 404 Group Not Found

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/***/Library/Python/3.7/bin//gitlabber", line 11, in <module>
    load_entry_point('gitlabber', 'console_scripts', 'gitlabber')()
  File "/Users/***/Library/Python/3.7/lib/python/site-packages/gitlabber/cli.py", line 43, in main
    tree.load_tree()
  File "/Users/***/Library/Python/3.7/lib/python/site-packages/gitlabber/gitlab_tree.py", line 158, in load_tree
    self.load_gitlab_tree()
  File "/Users/***/Library/Python/3.7/lib/python/site-packages/gitlabber/gitlab_tree.py", line 142, in load_gitlab_tree
    self.get_projects(group, node)
  File "/Users/***/Library/Python/3.7/lib/python/site-packages/gitlabber/gitlab_tree.py", line 105, in get_projects
    projects = group.projects.list(as_list=False, archived=self.archived)
  File "/Users/***/Library/Python/3.7/lib/python/site-packages/gitlab/exceptions.py", line 281, in wrapped_f
    raise error(e.error_message, e.response_code, e.response_body) from e
gitlab.exceptions.GitlabListError: 404: 404 Group Not Found

I suggest to modify in your PR following lines:

    def get_projects(self, group, parent):
        try:
            projects = group.projects.list(as_list=False, archived=self.archived)
            self.progress.update_progress_length(len(projects))
            self.add_projects(parent, projects)
        except GitlabListError as error:
            log.error(f"Error getting projects on {group.name} (ID: {group.id}): {error} )")

Can you please adapt in PR?

[odin568]

Will also test this evening, sorry für late reply

[odin568]

yes, have the same problem as @devstek . Manually applied the same code as suggested (replace of get_projects() - then it works for me too.

[devstek]

So i've made a pull request to @fmartingr forked project Link as it seems to work for others.

[fmartingr]

So i've made a pull request to @fmartingr forked project Link as it seems to work for others.

I've merged your PR, thanks for that! I'm sorry this is taking some time, but it's difficult to catch on my side since I cannot reproduce it.

[odin568]

@ezbz Any chance to get this merged and a new version to be released? Seems quite some people are affected. Can work with installation from sources for meantime, but when trying to enable other people use it as well, would be better to have the published version fixed. Thanks a lot!

[odin568]

Hi @ezbz sorry to bother again - this PR fixes my issue which still exist after several gitlab upgrades in the meantime. Any chance for getting merged and released? Makes it way easier to get directly via pip Thank you very much!

[devstek]

+1

[JustinPullen]

Would really love to use this tool just dont want to use my own fork. Can we get this merged?

[tewe]

Thanks, this fixed #49 for me. But adding min_access_level to some GitLab requests would be a smaller change and run faster.