search

open-telemetry / opentelemetry-collector-contrib

Contrib repository for the OpenTelemetry Collector
https://opentelemetry.io
Apache License 2.0
3.12k stars 2.39k forks source link

New component: GitLab Receiver #35207

Open adrielp opened 2 months ago

adrielp commented 2 months ago

The purpose and use-cases of the new component

The purpose of this receiver is the same as the GitHub Receiver but in the context of GItLab. It will support metrics, logs, and traces from GitLab in almost the same way as the GitLab receiver. Much of this code exists in some form or another.

Really, the logic and metrics are the same as the GitHub receiver. The differences are:

Everything else is closely aligned. The CICD SIG will leverage this as one of the prototypes alongside the GitHub receiver helping to inform conventions and support telemetry in GitLab.

Example configuration for the component

extensions:
  bearertokenauth/gitlab:
    token: ${env:TOKEN}

receivers:
  gitlab:
    interval: 15s #the interval at which the scraper scrapes
    group: mygroup #the organization name in github
    query: <some query>
    traces:
        .... <traces webhook configuration>
    auth:
      authenticator: bearertokenauth/gitlab

Telemetry data types supported

Metrics, Logs, Traces

Is this a vendor-specific component?

Code Owner(s)

adrielp

Sponsor (optional)

@atoulme

Additional context

No response

atoulme commented 1 month ago

I will sponsor this.

niwoerner commented 2 weeks ago

Hi all, thank you for all the great work you've been doing in context of CI/CD pipelines and OpenTelemetry.

I've been playing around with a Gitlabreceiver to create traces for CI/CD pipelines. That isn't exactly aligning with this issue, but since the Gitlabreicever was working pretty well for me, it might be interesting for more folks here in the community.

I'll be giving a talk about this next week at the Kubecon. If someone is present there, please let me know and we can catch up about this topic.

I would be happy to contribute (and maintain) this, if there would be an interest from the community. :)

adrielp commented 2 weeks ago

Hey @niwoerner , thanks for coming from LinkedIn and posting here on the issue! I think the alignment is quite nice, given the concept on the tracing end is the same. Would love to get support in contributing and maintaining this going forward. Let's sync up & figure out how we can move forward on this!

abitrolly commented 2 weeks ago

interval: 15s #the interval at which the scraper scrapes

I believe GitLab CLI can subscribe to events from running pipeline.

glab pipeline status --live

#  -l, --live            Show status in real time until the pipeline ends.

So maybe it is possible to receive events using some kind of (web)hook instead of the scraper.

The purpose of this receiver is the same as the https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/22028 but in the context of GItLab. It will support metrics, logs, and traces from GitLab in almost the same way as the GitLab receiver.

I would appreciate the expanded use cases without redirections to GitHub that may significantly drift away from what is it possible specifically with GitLab.

I will sponsor this.

What pain point do you want to address first?

adrielp commented 2 weeks ago

@abitrolly - the GitHub receiver is important context. Originally it was named the gitprovider receiver which I originally developed as a prototype here. It scraped both GitLab & GitHub's respective API's to build out a set of VCS Metrics.

Upon reconsideration of the implementation & incorporating other signals (tracing, logs), we as a community decided to separate things out and rename the receiver to be github receiver. While there are implementation differences, the high level concept, and core logic, is much the same. Events(logs) & Traces through Webhook + a set of metrics through scraping.

This is a good summary of the decisions and context I mentioned above.

abitrolly commented 2 weeks ago

@adrielp I am asking about specific use cases to estimate the scope of work left from https://github.com/niwoerner/gitlabreceiver. Ideally it should be a checkbox list.

niwoerner commented 2 weeks ago

I plan to sync with @adrielp this week at the Kubecon. (or virtually afterwards)

That would probably help to determine how to proceed further in regards to the tracing functionality of my created Gitlabreceiver. I would love to hear a second opinion on a few of my implementation choices.

Based on that outcome we could define next steps if (and how) we want to add the functionality to the Giltabreceiver here

PS: If anyone else reading this, is attending the upcoming Kubecon and would like to exchange about this topic, please let me know :)

abitrolly commented 2 weeks ago

@niwoerner I am not able to attend Kubecon financially, but I found your slides https://static.sched.com/hosted_files/kccncna2024/86/Kubecon-NA-2024-Gitlab-CICD-Pipelines-Otel.pdf :D

For some reason page with blue stripe background are very slow to render, like page 22. Maybe it is just a problem with my Firefox viewer.

  1. Is it possible to quickly see data from public projects like https://gitlab.com/gitlab-org/gitlab-development-kit/ without configuring anything?

  2. What software is used for The Pipeline Fitness Tracker with expected and actual duration?

niwoerner commented 2 weeks ago

@abitrolly

Regarding 1:

To get the traces for pipelines with the Gitlabreceiver you need to:

1.) have a custom otel collector with the Gitlabreceiver running somewhere

2.) Configure the group/repo webhook settings to send pipeline events to the collector.

Regarding 2:

Do you mean the notification slide?

The notification is received in Microsoft teams and sent from Dynatrace

Feel free to open a discussion in the Gitlabreceiver repo or send me a message via slack/linkedin if you have more questions about the presentation :)

(I don't want to abuse this issue for it :D)

abitrolly commented 2 weeks ago

@niwoerner

1) I guess the most simple collector to run for the demo is https://github.com/CtrlSpice/otel-desktop-viewer

2) In theory it is possible to hack glab ci status --live to get steam events without web hooks https://gitlab.com/gitlab-org/cli/-/blob/main/commands/ci/status/status.go?ref_type=heads#L42

Feel free to open a discussion in the Gitlabreceiver repo or send me a message via slack/linkedin if you have more questions about the presentation :)

Sure, but I don't think it hurts to discuss how to test current state of things with GitLab tracing in the most easy way here. )