search

MeltanoLabs / tap-gitlab

Singer.io Tap for extracting data from Gitlab's API
GNU Affero General Public License v3.0
8 stars 25 forks source link

Extract Merge Requests that will close an issue #18

Open pnadolny13 opened 2 years ago

pnadolny13 commented 2 years ago

In GitLab by @iroussos on Nov 12, 2019, 07:13

We currently export both the Issues and the Merge Requests for a Project.

We are not extracting the relation between Issues and Merge Requests that will close it.

That relation is important because it ties MRs to Issues and is also useful for tracking Cycle Analytics: From the moment an issue is created, to planning the issue, working on an issue in an MR, reviewing the MR and finally merging the MR and closing the issue.

This relation can be fetched in two ways:

  1. API call to list merge requests that will close issue on merge
  2. API call to List issues that will close on merge of an MR

We can use either one, but we have to be careful:

Similarly to fetching the Commits for a Merge Request, this operation requires one API call per Merge Request (or Issue), so going through that flow can slow down considerably the end-to-end extraction time. For example, in a project like gitlab-org/gitlab-foss, this would result to 15x more API calls than required for fetching all the other Entities supported by tap-gitlab.

The approach should be similar to what we did with fetching the Commits for a Merge Request, with a configuration parameter similar to fetch_merge_request_commits that will allow users to manually set this option for projects they know can handle that type of N+1 querying.

Useful Resources when working with Gitlab's API:

  1. Gitlab API Documentation
  2. Gitlab API entities
  3. Entity Schemas
pnadolny13 commented 2 years ago

In GitLab by @iroussos on Nov 12, 2019, 07:14

removed milestone

pnadolny13 commented 2 years ago

In GitLab by @iroussos on Nov 12, 2019, 07:17

changed title from Ex{-port and create table for-} Merge Requests that will close an issue to Ex{+tract+} Merge Requests that will close an issue

pnadolny13 commented 2 years ago

In GitLab by @iroussos on Nov 12, 2019, 07:27

changed the description

pnadolny13 commented 2 years ago

In GitLab by @iroussos on Nov 12, 2019, 08:18

changed the description

pnadolny13 commented 2 years ago

In GitLab by @iroussos on Nov 12, 2019, 08:21

A followup when this issue is completed can be to add the Transformations necessary for this new relation table.

This can be done in the dbt package with the Transformations for gitlab data in a similar way as described in meltano/dbt-tap-gitlab#5.