List project submodules and get the commits they point to with python-gitlab.
The Gitlab REST API V4 doesn't implement anything for submodule inspection yet. The only thing we can currently do is updating a submodule to a new commit id, but how can we trigger such an update if we don't know the current commit id of our submodule?
If you're using python-gitlab
and you're distributing shared code among
your projects with submodules, you've probably run into this issue already.
This package provides minimal utils to list the submodules present in a Gitlab project, and more importantly to get the commits they're pointing to.
Internally, it reads and parses the .gitmodules
file at the root of the
Project. To get the commit id of a submodule, it finds the last commit that
updated the submodule and parses its diff (this can sometimes fail due to a
limit of the GitLab API itself -
see Limitations).
About the future of this package
I don't plan to make PRs to python-gitlab
for now.
In my opinion this problem should ideally be fixed in the Gitlab REST API,
and then python-gitlab
could wrap around the new endpoints.
So I see this package as a temporary solution until the API gets extended with more submodule functionalities.
@darkdragon-001 created an issue on GitLab about the lack of support for submodules, feel free to support it with a thumb up: https://gitlab.com/gitlab-org/gitlab/-/issues/352836
python-gitlab
since version 3.0.0
)pip install python-gitlab-submodule
or directly from Github:
pip install git+git://github.com/ValentinFrancois/python-gitlab-submodule#egg=python-gitlab-submodule
from gitlab import Gitlab
from gitlab_submodule import iterate_subprojects
gl = Gitlab() inkscape = gl.projects.get('inkscape/inkscape') subprojects = iterate_subprojects( inkscape, gl,
ref='e371b2f826adcba316f2e64bbf2f697043373d0b')
for subproject in subprojects: print('- {} ({}) -> {}'.format( subproject.submodule.path, subproject.project.web_url, subproject.commit.id if subproject.commit else '?'))
Output:
for subproject in subprojects:
Output:
iterate_subprojects(...)
What you'll probably use most of the time.
Subproject
objects that describe the submodules.
iterate_subprojects(
project: Project,
gl: Union[Gitlab, ProjectManager],
ref: Optional[str] = None,
only_gitlab_subprojects: bool = False,
self_managed_gitlab_host: Optional[str] = None
) -> Generator[Subproject, None, None]
project
: a gitlab.v4.objects.Project
objectgitlab
: the gitlab.Gitlab
instance that you used to authenticate, or its
projects: gitlab.v4.objects.ProjectManager
attributeref
: (optional) a ref to a branch, commit, tag etc. Defaults to the
HEAD of the project default branch.only_gitlab_subprojects
: (optional) if set to True
, will ignore the
submodules not hosted on GitLab. If set to False
(default), it will yield
Subproject
objects with self.project = None
for submodules not hosted on GitLab.self_managed_gitlab_host
: (optional) if some submodules are hosted on a
self-managed GitLab instance, you should pass its url here otherwise it
may be impossible to know from the URL that it's a GitLab project.Generator of Subproject
objects
./<submodule_dir>
,
in some rare cases ./<submodule_dir>
might not be part of the diff
object returned by the GitLab API. In that case we have no other choice than
set Subproject.commit
to None
, that's why the two examples above
check if subproject.commit
is not None
before using the value
subproject.commit.id
.list_subprojects(...)
Same parameters as iterate_subprojects(...)
but
returns a list
of Subproject
objects.
Subproject
Basic objects that contain the info about a Gitlab subproject.
project: Optional[gitlab.v4.objects.Project]
: the Gitlab project that the
submodule links to (can be None
if the submodule is not hosted on GitLab)submodule:
Submodule
: a basic object that contains
the info found in the .gitmodules
file (name, path, url).commit: Union[gitlab.v4.objects.ProjectCommit, Commit]
: the commit that
the submodule points to (if the submodule is not hosted on GitLab, it will
be a dummy Commit
object with a single attribute id
)str()
output:<class 'Subproject'> => {
'submodule': <class 'Submodule'> => {'name': 'share/extensions', 'parent_project': <class 'gitlab.v4.objects.projects.Project'> => {'id': 3472737, 'description': 'Inkscape vector image editor', 'name': 'inkscape', 'name_with_namespace': 'Inkscape / inkscape', 'path': 'inkscape', 'path_with_namespace': 'inkscape/inkscape', 'created_at': '2017-06-09T14:16:35.615Z', 'default_branch': 'master', 'tag_list': [], 'topics': [], 'ssh_url_to_repo': 'git@gitlab.com:inkscape/inkscape.git', 'http_url_to_repo': 'https://gitlab.com/inkscape/inkscape.git', 'web_url': 'https://gitlab.com/inkscape/inkscape', 'readme_url': 'https://gitlab.com/inkscape/inkscape/-/blob/master/README.md', 'avatar_url': 'https://gitlab.com/uploads/-/system/project/avatar/3472737/inkscape.png', 'forks_count': 900, 'star_count': 2512, 'last_activity_at': '2022-01-29T23:45:49.894Z', 'namespace': {'id': 470642, 'name': 'Inkscape', 'path': 'inkscape', 'kind': 'group', 'full_path': 'inkscape', 'parent_id': None, 'avatar_url': '/uploads/-/system/group/avatar/470642/inkscape.png', 'web_url': 'https://gitlab.com/groups/inkscape'}}, 'parent_ref': 'e371b2f826adcba316f2e64bbf2f697043373d0b', 'path': 'share/extensions', 'url': 'https://gitlab.com/inkscape/extensions.git'},
'project': <class 'gitlab.v4.objects.projects.Project'> => {'id': 5833962, 'description': 'Python extensions for Inkscape core, separated out from main repository.', 'name': 'extensions', 'name_with_namespace': 'Inkscape / extensions', 'path': 'extensions', 'path_with_namespace': 'inkscape/extensions', 'created_at': '2018-03-22T00:29:09.053Z', 'default_branch': 'master', 'tag_list': ['addin', 'additional', 'addon', 'core', 'extension', 'inkscape', 'python'], 'topics': ['addin', 'additional', 'addon', 'core', 'extension', 'inkscape', 'python'], 'ssh_url_to_repo': 'git@gitlab.com:inkscape/extensions.git', 'http_url_to_repo': 'https://gitlab.com/inkscape/extensions.git', 'web_url': 'https://gitlab.com/inkscape/extensions', 'readme_url': 'https://gitlab.com/inkscape/extensions/-/blob/master/README.md', 'avatar_url': 'https://gitlab.com/uploads/-/system/project/avatar/5833962/addons.png', 'forks_count': 89, 'star_count': 41, 'last_activity_at': '2022-01-29T19:10:13.502Z', 'namespace': {'id': 470642, 'name': 'Inkscape', 'path': 'inkscape', 'kind': 'group', 'full_path': 'inkscape', 'parent_id': None, 'avatar_url': '/uploads/-/system/group/avatar/470642/inkscape.png', 'web_url': 'https://gitlab.com/groups/inkscape'}},
'commit': <class 'gitlab.v4.objects.commits.ProjectCommit'> => {'id': '6c9b68507be427bffba23507bbaacf3f8a0f3752', 'short_id': '6c9b6850', 'created_at': '2021-11-28T22:23:47.000+00:00', 'parent_ids': ['fdda3f18b3ddda61a19f5046ce21a6e2147791f5', '8769b39a55f94d42ac0d9b24757540a88f2865cc'], 'title': "Merge branch 'add-issue-template-bug-report' into 'master'", 'message': "Merge branch 'add-issue-template-bug-report' into 'master'\n\nadd issue template for GitLab for bug reports\n\nSee merge request inkscape/extensions!377", 'author_name': 'Martin Owens', 'author_email': 'doctormo@geek-2.com', 'authored_date': '2021-11-28T22:23:47.000+00:00', 'committer_name': 'Martin Owens', 'committer_email': 'doctormo@geek-2.com', 'committed_date': '2021-11-28T22:23:47.000+00:00', 'trailers': {}, 'web_url': 'https://gitlab.com/inkscape/extensions/-/commit/6c9b68507be427bffba23507bbaacf3f8a0f3752', 'stats': {'additions': 25, 'deletions': 0, 'total': 25}, 'status': 'success', 'project_id': 5833962, 'last_pipeline': {'id': 417958828, 'iid': 924, 'project_id': 5833962, 'sha': '6c9b68507be427bffba23507bbaacf3f8a0f3752', 'ref': 'master', 'status': 'success', 'source': 'push', 'created_at': '2021-11-28T22:23:48.313Z', 'updated_at': '2021-11-28T22:31:49.083Z', 'web_url': 'https://gitlab.com/inkscape/extensions/-/pipelines/417958828'}, 'is_exact': True}
}
list_submodules(...)
Lists the info about the project submodules found in the .gitmodules
file.
list_project_submodules(
project: Project,
ref: Optional[str] = None) -> List[Submodule]
project
: a gitlab.v4.objects.Project
objectref
: (optional) a ref to a branch, commit, tag etc. Defaults to the
HEAD of the project default branch.list
of Submodule
objects
Submodule
Represents the .gitmodules
config of a submodule + adds info about the
parent project
parent_project: gitlab.v4.objects.Project
: project that uses the submoduleparent_ref: str
: ref where the .gitmodules
file was readname: str
: local name used by git for the submodulepath: str
: local path pointing to the submodule directory in the projecturl: str
: URL linking to the location of the repo of the submodule (not
necessarily Gitlab)str()
output:<class 'Submodule'> => {'name': 'share/extensions', 'parent_project': <class 'gitlab.v4.objects.projects.Project'> => {'id': 3472737, 'description': 'Inkscape vector image editor', 'name': 'inkscape', 'name_with_namespace': 'Inkscape / inkscape', 'path': 'inkscape', 'path_with_namespace': 'inkscape/inkscape', 'created_at': '2017-06-09T14:16:35.615Z', 'default_branch': 'master', 'tag_list': [], 'topics': [], 'ssh_url_to_repo': 'git@gitlab.com:inkscape/inkscape.git', 'http_url_to_repo': 'https://gitlab.com/inkscape/inkscape.git', 'web_url': 'https://gitlab.com/inkscape/inkscape', 'readme_url': 'https://gitlab.com/inkscape/inkscape/-/blob/master/README.md', 'avatar_url': 'https://gitlab.com/uploads/-/system/project/avatar/3472737/inkscape.png', 'forks_count': 900, 'star_count': 2512, 'last_activity_at': '2022-01-29T23:45:49.894Z', 'namespace': {'id': 470642, 'name': 'Inkscape', 'path': 'inkscape', 'kind': 'group', 'full_path': 'inkscape', 'parent_id': None, 'avatar_url': '/uploads/-/system/group/avatar/470642/inkscape.png', 'web_url': 'https://gitlab.com/groups/inkscape'}}, 'parent_ref': 'e371b2f826adcba316f2e64bbf2f697043373d0b', 'path': 'share/extensions', 'url': 'https://gitlab.com/inkscape/extensions.git'}
submodule_to_subproject(...)
Converts a Submodule
object to a Subproject
object, assuming it's
hosted on Gitlab.
Raises a FileNotFoundError
if the path of the submodule actually doesn't
exist in the host repo or if the url of the submodule doesn't link to an
existing repo (both can happen if you modify the .gitmodules
file without
using one of the git submodule
commands).
submodule_to_subproject(
gitmodules_submodule: Submodule,
gl: Union[Gitlab, ProjectManager],
self_managed_gitlab_host: Optional[str] = None,
) -> Subproject
Parameter details: See iterate_subprojects(...)
PRs are appreciated, just make sure your PR title starts with one of the following keywords so that the CI works:
[MAJOR]
: breaking changes[MINOR]
: feature changes[PATCH]
: fixes[CONFIG]
: changes only related to GitHub (CI, .gitignore, etc.) -> won't trigger a package release