search

MeltanoLabs / tap-gitlab

Singer.io Tap for extracting data from Gitlab's API
GNU Affero General Public License v3.0
11 stars 27 forks source link

Add support for incremental updating to Tap-GitLab #9

Closed pnadolny13 closed 2 years ago

pnadolny13 commented 2 years ago

In GitLab by @zj-gitlab on Jun 14, 2019, 04:42

For now I've limited my data to the gitlab-org group, and a small project in that group. Now when I would add gitlab-ce to the projects I fear the updating of the data would take very long. Making this tap less interesting to use for large projects or groups.

/cc @iroussos

pnadolny13 commented 2 years ago

In GitLab by @iroussos on Jun 14, 2019, 07:14

assigned to @iroussos

pnadolny13 commented 2 years ago

In GitLab by @iroussos on Jun 20, 2019, 14:21

mentioned in merge request !14

pnadolny13 commented 2 years ago

In GitLab by @iroussos on Jun 20, 2019, 14:46

closed via merge request !14

pnadolny13 commented 2 years ago

In GitLab by @iroussos on Jun 20, 2019, 14:46

mentioned in commit 702dde87c04c72be83eb0750fd341c3a1a56784a

pnadolny13 commented 2 years ago

In GitLab by @iroussos on Jun 20, 2019, 14:46

@zj all the issues you had should have been fixed with the latest updates. Check meltano/tap-gitlab!14 for more info.

The update speeds up both the initial execution of the Tap and all subsequent executions by properly using both the start date and the STATE of the previous run.

I just extracted one month of data from gitlab-org/gitlab-ce in ~30 minutes, including all the Epics for gitlab-org, the overhead from extracting the labels for the project and the Group and the fact that a lot of issues (4480), MRs (1763) and Epics (~500) are marked as updated during the last 20 days.

pnadolny13 commented 2 years ago

In GitLab by @iroussos on Jun 20, 2019, 15:05

changed title from {-Tap-GitLab doesn't support incremental updating-} to {+Add support for incremental updating to Tap-GitLab+}