[GitLab] Unable to find repository for group (404) #10

Closed szpak closed 1 month ago

Plying with a PAT token generated for one project and for one group works fine if a particular repos are pointed in REPOS. However, if for the same group token, I point test-group/test-subgroup where a few projects I would like to analyze are located, I've got 404.

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/gitlab/exceptions.py", line 340, in wrapped_f
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/gitlab/mixins.py", line 123, in get
    server_data = self.gitlab.http_get(path, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/gitlab/client.py", line 827, in http_get
    result = self.http_request(
             ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/gitlab/client.py", line 793, in http_request
    raise gitlab.exceptions.GitlabHttpError(
gitlab.exceptions.GitlabHttpError: 404: 404 Project Not Found

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/conf.py", line 83, in load_and_verify_configuration
    configuration.repos.append(scm_client.get_repository(owner_or_repo))
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/client_impl/gitlab_client.py", line 27, in get_repository
    project = self._gitlab_client.projects.get(owner_and_name)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/gitlab/v4/objects/projects.py", line 787, in get
    return cast(Project, super().get(id=id, lazy=lazy, **kwargs))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/gitlab/exceptions.py", line 342, in wrapped_f
    raise error(e.error_message, e.response_code, e.response_body) from e
gitlab.exceptions.GitlabGetError: 404: 404 Project Not Found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/main.py", line 11, in <module>
    configuration = load_and_verify_configuration()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/conf.py", line 85, in load_and_verify_configuration
    raise ValueError(f"Unable to find repository {owner_or_repo}, aborting: {e}")
ValueError: Unable to find repository test-group/test-subgroup, aborting: 404: 404 Project Not Found
datascraper exited with code 1

(REPOS=test-group/test-subgroup)

@robrobert99 Have you played with the groups in your environment? Is it supported for GitLab?

I took a closer look at that problem and there seem to be more than one place to enhance.

"/" is not necessarily an indication that the value is a repository. GitLab allows subgroups which can be handy in the corporation environment. E.g. department1/teamA/projectX, department2/teamT/projectY:

https://github.com/MShekow/renovate-pr-visualization/blob/63078a0b71abe4ed40e1dc88cf5caf7b5544e245/conf.py#L81-L87

In my case it tried to get a project based on the subgroup name.

Even with 1 fixed, the related problem occurs in:

https://github.com/MShekow/renovate-pr-visualization/blob/63078a0b71abe4ed40e1dc88cf5caf7b5544e245/client_impl/gitlab_client.py#L88

Configuration check successful, starting to fetch Renovate PRs (this may take a few minutes) ...
Traceback (most recent call last):
  File "/app/main.py", line 13, in <module>
    renovate_prs = get_renovate_prs(configuration)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/renovate_parser.py", line 78, in get_renovate_prs
    for pr in git_repo.get_pull_requests(pr_author_username=config.renovate_scm_user, renovate_pr_label=config.renovate_pr_label):
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/client_impl/gitlab_client.py", line 88, in get_pull_requests
    for mr in self._gl_project.mergerequests.list(**mr_params):
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/gitlab/base.py", line 134, in __getattr__
    raise AttributeError(message)
AttributeError: 'GroupProject' object has no attribute 'mergerequests'

<class 'gitlab.v4.objects.projects.GroupProject'> was created via a
list() call and only a subset of the data may be present. To ensure
all data is present get the object using a get(object.id) call. For
more details, see:

https://python-gitlab.readthedocs.io/en/v4.6.0/faq.html#attribute-error-list

A group is fetched which obviously doesn't have merge requests.

There could be some extra logic in get_repository() to recursively fetch projects from group subgroups, but I didn't like the scratchy implementation I did (I'm not a Python developer), so I paste here only pseudo code:

    def get_repositories(self, owner_or_username: str) -> List[GitRepository]:
        if owner_or_username.startswith("user:"):
            owner_or_username = owner_or_username[5:]
            if self.get_username().lower() == owner_or_username.lower():
                projects = self._gitlab_client.projects.list(owned=True)
            else:
                user = self._gitlab_client.users.list(username=owner_or_username)[0]
                projects = self._gitlab_client.projects.list(user_id=user.id)
        else:
            group = self._gitlab_client.groups.get(owner_or_username)
            projects = self.get_repositories_from_subgroup_recursively(group)

        repos: List[GitRepository] = [GitLabRepository(project.id, self._gitlab_client, project) for project in
                                      projects]

        return repos

    def get_repositories_from_subgroup_recursively(self, group: Group) -> List[GitRepository]:
        projects = group.projects.list()
        sub_groups = group.subgroups.list()

        sub_projects = []
        for sub_group in sub_groups:
            sub_projects += self.get_repositories_from_subgroup_recursively(todo_create_domain_group_object_from_rest_group_object(sub_group))

        return projects + sub_projects

hoping that you will figure out any better solution to handle that :-/

@robrobert99 @MShekow WDYT?

Closed in https://github.com/MShekow/renovate-pr-visualization/pull/11

MShekow / renovate-pr-visualization

[GitLab] Unable to find repository for group (404) #10