search

MeltanoLabs / tap-gitlab

Singer.io Tap for extracting data from Gitlab's API
GNU Affero General Public License v3.0
11 stars 27 forks source link

Extract Comments for Issues, Merge Requests and Epics #19

Open pnadolny13 opened 2 years ago

pnadolny13 commented 2 years ago

In GitLab by @iroussos on Nov 12, 2019, 07:27

We are currently not extracting the Comments on Issues and Merge Requests.

All comments can be fetched by using the notes API end point:

  1. List project issue notes
  2. List merge request notes
  3. List epic notes (should be enabled only if "ultimate_license": true)

Similarly to fetching the Commits for a Merge Request, this operation requires one API call per Merge Request (or Issue / Epic), so going through that flow can slow down considerably the end-to-end extraction time. For example, in a project like gitlab-org/gitlab-foss, this would result to an additional ~90K more API calls.

The approach should be similar to what we did with fetching the Commits for a Merge Request, with a configuration parameter similar to fetch_merge_request_commits that will allow users to manually set this option for projects they know can handle that type of N+1 querying.

Useful Resources when working with Gitlab's API:

  1. Gitlab API Documentation
  2. Gitlab API entities
  3. Entity Schemas
pnadolny13 commented 2 years ago

In GitLab by @iroussos on Nov 12, 2019, 07:28

changed title from Extract Comments {-(Notes) -}for Issues, Merge Requests and Epics to Extract Comments for Issues, Merge Requests and Epics

pnadolny13 commented 2 years ago

In GitLab by @iroussos on Nov 12, 2019, 08:21

A followup when this issue is completed can be to add the Transformations necessary for those three new tables.

This can be done in the dbt package with the Transformations for gitlab data.