Pull in Epics from the GitLab.com API #6

Closed pnadolny13 closed 2 years ago

In GitLab by @NicoleSchwartz on Jun 12, 2019, 11:18

Context: Nicole reached out because she is a product manager and, in her own words, "want to build better future backlog grouping / viz".

Opportunity assessment: https://docs.google.com/document/d/1YiZmzVkcCTmpZX7IjIXN-hOzv-IQmWgD5FK8ulTCu4o/edit#heading=h.wtfp7g1r16nq

In GitLab by @dmor on Jun 12, 2019, 11:19

assigned to @dmor

In GitLab by @dmor on Jun 12, 2019, 11:19

changed title from Pull in Epics from the GitLab.com API to {+Exploratory Sprint: +}Pull in Epics from the GitLab.com API

In GitLab by @dmor on Jun 12, 2019, 11:20

assigned to @NicoleSchwartz

In GitLab by @dmor on Jun 12, 2019, 11:21

changed the description

In GitLab by @dmor on Jun 12, 2019, 11:22

changed the description

In GitLab by @NicoleSchwartz on Jun 12, 2019, 11:26

As a product manager with a large backlog of both epics and issues i would love to be able to create a dashboard to view the following:

epics and their related issues grouped
epics and or issues grouped by milestone in a "roadmap" filtered by specific labels

Epic grouping would be nice to be able to tell

how many epics are what % complete
how many issues are not grouped to an epic and what their labels/themes are
how many epics have no children and what their labels/themes are
find themes for issues (labels) and perhaps ID trends i need to research or create epics for
see if some issues need to be added to an epic or have an epic created

Roadmap

"roadmap" epics and issues for upcoming releases
"roadmap" epics and issues based on themes / teams - i want to see my whole stage which has multiple groups

Overall I want to be able to view the large amount of backlog and look for trends, orphans, groups, new items and if they have commonalities

In GitLab by @NicoleSchwartz on Jun 12, 2019, 11:27

Here is a board UX is creating and having to maintain by hand: https://app.mural.co/t/gitlab2474/m/gitlab2474/1559760279509/a3bb0208e7eb3dce96a6b6de13ec9ab4631029e1

In GitLab by @dmor on Jun 12, 2019, 11:30

assigned to @iroussos

In GitLab by @NicoleSchwartz on Jun 12, 2019, 11:31

in my mind mind-map or connectivity graphs - https://images.app.goo.gl/q6V2Ry4AMU3HjoSe6 would be a good way to find the orphans/trends (i could be wrong and am open to other ways), possibly by labels (filter in some, filter out some)

and for the roadmaps something more linear and progressive (by a field, i would pick milestone) and not as crazy connected (but i still want to be able to drill epic to issues) but grouped by things (perhaps groups, perhaps stages, etc)

In GitLab by @iroussos on Jun 13, 2019, 09:55

marked this issue as related to meltano#708

In GitLab by @dmor on Jun 13, 2019, 14:04

unassigned @dmor

In GitLab by @iroussos on Jun 14, 2019, 08:42

mentioned in merge request !11

In GitLab by @iroussos on Jun 14, 2019, 11:14

Fetching Epics and the issues for each Epic has been added to tap-gitlab.

I am keeping this issue open in order to track the overall progress:

I am going to work on adding some advanced Transforms and Meltano Models to try and support what @NicoleSchwartz has requested. Not all features can be readily available in our current version of Meltano UI, but I am going to try to have as much as possible there.

In GitLab by @iroussos on Jun 17, 2019, 11:26

marked this issue as related to meltano#717

In GitLab by @iroussos on Jun 18, 2019, 08:22

marked this issue as related to model-gitlab-ultimate#1

In GitLab by @iroussos on Jun 18, 2019, 10:46

Transforms and Meltano Models for supporting Epics and the requested analysis have been added.

Because Meltano UI is still a work in progress and does not yet support filtering and multi attribute ordering, I have also created a Jupyter Notebook that can provide a way to access the data.

I will add more details tomorrow morning, with the available options and how we could move forward.

In GitLab by @iroussos on Jun 19, 2019, 07:40

@NicoleSchwartz a quick summary of what's there after this first iteration:

Most of the data you want to see are fetched. Even though we can not currently automatically produce the fancy reports you may need, I hope that you can use the data fetched to speed up your process and explore what's there.

I am going to use the data from the Meltano Group to showcase the available options. We are not very heavy users of the Epics feature and we are currently iterating very quickly with one week milestones, so some of the screenshots may be poor with respect to what is shown, but you can imagine how they would look with more issues on epics.

You can get a quick overview of the progress of the open Epics by using the gitlab_epics_overview table:

Or you can see open Epics with no children, check their labels, etc:

You can use the gitlab_issues_per_epic table to check open Epics and all their related issues by milestone and filter by issue status (keep only the open ones), milestone or issue label (and much more).

For example, in the following I have only kept the open issues of the open Epics and grouped them together and ordered them by milestone. I have also filtered the result to only keep issues labeled as "To Do"

I have kept the backlog and ideas open ended milestones, as we don't have that many issues in Epics at the moment, but the idea is that this report would allow you to check the open issues for an epic (e.g. "Exploratory Sprints") for the next few milestones. If I can understand correctly, this covers half the data you need for generating the roadmap board you are currently using.

The other half are issues not in Epics, and you can get those from the gitlab_open_issues_not_in_epic table. For example, in the following I have selected all the open issues not in an epic (together with their labels) for the next couple milestones in Meltano:

From there, as we have all the data for a group, we can do whatever else we may want. For example, we could generate the top N most popular labels on open issues not in an Epic, ordered by the number of issues that are labeled as such:

'No Label' is a special placeholder we use for issues with no label at all, which shows that a lot of issues in Meltano at the moment are not labeled. Also, we can see that we have a lot of "Documentation" issues not in an Epic, so maybe we should create one to track them more effectively.

In GitLab by @iroussos on Jun 19, 2019, 08:09

Next steps if you want to check everything I mentioned:

(1) Install and learn how to use meltano

You should check our Installation and Getting Started Guides

(2) Fetch your data from Gitlab.com

I think that we do not currently have a tutorial for fetching data from Gitlab (am I right @bencodezen?), so I am going to try to help with what you have to do.

You should create a new project and use Meltano UI by following the "Getting Started Guide":

Add the tap-gitlab extractor
The settings for the extractor should be as follows:
- api_url --> leave it as it is
- groups --> Groups to track (space separated), e.g. meltano or gitlab-org
- projects --> Projects to track (space separated), e.g. gitlab-ce
- private_token --> your GitLab access token (gitlab.com > profile page > Create an access token)
- start_date --> earliest day data will be brought in for (as a timestamp). e.g. 2019-03-01T00:00:00Z. (we will soon update this to work better)
- ultimate_license --> true (as you have one and it enables fetching epics)
Notes on group and project options:
- either groups or projects need to be provided
- filling in 'groups' but leaving 'projects' empty will sync all group projects.
- filling in 'projects' but leaving 'groups' empty will sync selected projects.
- filling in 'groups' and 'projects' will sync selected projects of those groups.
Add the target-postgres Loader and fill the settings with your local postgres setup.
Run the elt with transforms enabled.

Or you can do everything on the cli if you prefer working outside of a UI:

meltano init nicoles-gitlab-project
cd nicoles-gitlab-project

# Update the .env in that folder with what follows

meltano add extractor tap-gitlab
meltano add loader target-postgres
meltano add model model-gitlab
meltano add model model-gitlab-ultimate
source .env
meltano elt tap-gitlab target-postgres --transform run

.env

export FLASK_ENV=development
export SQLITE_DATABASE=meltano

export PG_DATABASE=YOUR_POSTGRES_DB_NAME
export PG_PASSWORD=YOUR_POSTGRES_PASSWORD
export PG_USERNAME=YOUR_POSTGRES_USERNAME
export PG_ADDRESS=localhost
export PG_PORT=5432
export PG_SCHEMA='tap_gitlab'

export GITLAB_API_TOKEN=' Your Gitlab API token'
export GITLAB_API_GROUPS='meltano'
export GITLAB_API_PROJECTS=''
export GITLAB_API_START_DATE='2018-01-01T00:00:00Z'
export GITLAB_API_ULTIMATE_LICENSE="true"

In GitLab by @iroussos on Jun 19, 2019, 08:19

Finally, you have two options for analysing your data:

use meltano ui, select the analyze tap and then one of the options available for Gitlab and Gitlab Ultimate (that's where the Epics are)
use the Jupyter Notebook I am going to attach if you prefer Jupyter. Jupyter is a less user friendly, way more technical option, but allows you to do way more powerful reporting. Hopefully we'll catchup and make Meltano UI as powerful, but we are a long way from the raw power of running any sql you want.

You can check the Using Jupyter Notebooks Tutorial if you want to see how to use the Jupyter Notebook.

Gitlab_Analytics_Ultimate.ipynb

In GitLab by @iroussos on Jun 19, 2019, 08:45

changed title from {-Exploratory Sprint: -}Pull in Epics from the GitLab.com API to Pull in Epics from the GitLab.com API

In GitLab by @bencodezen on Jun 19, 2019, 09:05

@iroussos There is a tutorial! Sorry for it not being as discoverable: https://www.meltano.com/docs/tap-gitlab-tutorial.html

I'll make sure to fix that this week.

In GitLab by @iroussos on Jun 19, 2019, 09:29

That's amazing @bencodezen! Your explanation of the various steps rocks! I wish I can write like you when I grow up :-)

We should also add the new option ("ultimate_license": true / false) for enabling the tap to fetch the entities that are only available for Gitlab Ultimate and Gitlab.com Gold accounts. (Maybe a section dedicated to those accounts?)

Also, we should not forget to add the --transform run flag on the meltano elt, so that the transforms run.

And the two models that users should add (the second only for the Gitlab Ultimate and Gitlab.com Gold accounts):

meltano add model model-gitlab
meltano add model model-gitlab-ultimate

In GitLab by @bencodezen on Jun 19, 2019, 11:57

@iroussos You are too kind! I've gone ahead and create an issue here: https://gitlab.com/meltano/meltano/issues/729

In GitLab by @NicoleSchwartz on Jun 19, 2019, 19:31

awesome thank you, hopefully i can start by easily getting the data via more complex queries than i can today!

In GitLab by @iroussos on Jun 20, 2019, 15:03

A quick note: We just released a major update for tap-gitlab that makes incrementally extracting from large projects like gitlab-ce / gitlab-ee way faster and more reliable.

If you have already created a Meltano project, please run the following command: meltano install

In GitLab by @iroussos on Jun 21, 2019, 05:28

I am closing this issue as we have successfully added support for extracting Epics and Epic Issues on tap-gitlab.

We have also supported those extracted entities by adding advanced transforms that generate tables to be used for analysis and have added .m5o models for accessing epics in various ways through Meltano UI.

If we need additional transforms and/or .m5o models, we can create new issues in the meltano project and continue the discussion there.

In GitLab by @iroussos on Jun 21, 2019, 05:28

closed

MeltanoLabs / tap-gitlab

Pull in Epics from the GitLab.com API #6