liatrio-otel-collector

The Liatrio OTEL Collector is an upstream distribution of the Open Telemetry collector, rebuilt with custom packages hosted within this repository. These custom packages are by default targeted for downstream contribution to Open Telemetry; pursuant acceptance by the community.

Quick Start Guide

Before diving into the codebase, you'll need to set up a few things. This guide is designed to help you get up and running in no time!

Here are the steps to quickly get started with this project:

Install Go: Use Homebrew to install Go with the following command:
```
brew install go
```
Install pre-commit: With the help of Homebrew, install the pre-commit as follows:
```
brew install pre-commit
```
Initialize pre-commit: Once installed, initiate pre-commit by running:
```
pre-commit install
```
Set up GitHub and/or GitLab scraper(s).
Start the collector: Finally, initiate the program:
```
make run
```

Configure GitHub Scraper (using Github Auth Extension)

To configure the GitHub Scraper you will need to make the following changes to config/config.yaml:

1) Uncomment/add extensions.basicauth/github section

```yaml
basicauth/github:
    client_auth:
        username: ${env:GITHUB_USER}
        password: ${env:GITHUB_PAT}
```

2) Uncomment/add receivers.gitprovider.scrapers.github.auth section

```yaml
auth:
    authenticator: basicauth/github
```

3) Add basicauth/github to the service.extensions list

```yaml
extensions: [health_check, pprof, zpages, basicauth/github]
```

4) Set environment variables: GITHUB_ORG, GITHUB_USER, and GITHUB_PAT

Configure GitHub Scraper

To configure the GitHub Scraper you will need to make the following changes to config/config.yaml:

1) Uncomment/add extensions.bearertokenauth/github section

```yaml
bearertokenauth/github:
    token: ${env:GH_PAT}
```

2) Uncomment/add receivers.github.scrapers.github.auth section

```yaml
auth:
    authenticator: bearertokenauth/github
```

3) Add basicauth/github to the service.extensions list

```yaml
extensions: [basicauth/github]
```

4) Set environment variables: GH_ORG, and GH_PAT

Configure GitLab Scraper

To configure the GitLab Scraper you will need to make the following changes to config/config.yaml

1) Uncomment/add extensions.bearertokenauth/gitlab section

```yaml
bearertokenauth/gitlab:
    token: ${env:GITLAB_PAT}
```

2) Uncomment/add receivers.gitlab.scrapers.gitlab.auth section

```yaml
auth:
    authenticator: bearertokenauth/gitlab
```

3) Add bearertokenauth/gitlab to the service.extensions list

```yaml
extensions: [bearertokenauth/gitlab]
```

4) Set environment variables: GL_ORG and GITLAB_PAT

Exporting to Grafana Cloud

If you want to export your data to Grafana Cloud through their OTLP endpoint, there's a couple of extra things you'll need to do.

Run export GRAF_USER and export GRAF_PAT with your instance id and cloud api key
Update the config/config.yaml file with the following:

extensions:
  ...

  basicauth/grafana:
    client_auth:
      username: ${env:GRAF_USER}
      password: ${env:GRAF_PAT}
...

exporters:
  ...
  otlphttp:
    auth:
      authenticator: basicauth/grafana
    endpoint: https://otlp-gateway-prod-us-central-0.grafana.net/otlp

...

service:
  extensions: [..., basicauth/grafana]
  pipelines:
    metrics:
      receivers: [otlp, gitprovider]
      processors: []
      exporters: [..., otlphttp]

Debugging

To debug through vscode:

1) Create a .debug.env file in the root of the repo, make a copy of the example

2) run make build-debug

3) run cd build && code .

4) With your cursor focused in main.go within the build directory. Launch the Launch Otel Collector in debug" debug configuration.

OTEL Intro

OTEL is a protocol used for distributed logging, tracing, and metrics. To collect metrics from various services, we need to configure receivers. OTEL provides many built-in receivers, but in certain cases, we may need to create custom receivers to meet our specific requirements.

A custom receiver in OTEL is a program that listens to a specific endpoint and receives incoming log, metrics, or trace data. This receiver then pipes the content to a process, for it to then be exported to a data backend.

Creating a custom receiver in OTEL involves implementing the receiver interface and defining the endpoint where the receiver will listen for incoming trace data. Once the receiver is implemented, it can be deployed to a specific location and configured to start receiving trace data.

Prereqs

There is currently a guide to build a custom trace receiver. It is a long read, requires a fairly deep understanding, and is slightly out of date due to non-backwards compatible internal API breaking changes. This document and receiver example attempts to simplify that to an extent.

There are a few main concepts that should help you get started:

Get familiar with the ocb tool. It is used to build custom collectors using a build-config.yaml file.
Get familiar with Go & the Factory design pattern.
Clearly define what outcome you want before building a customization.
Get familiar with Go interfaces.
Get familiar with delv the go debugger.

liatrio / liatrio-otel-collector

readme