Discussion: The complete guide to OpenTelemetry in Golang.

[komuw]

This is the place for comments/discussion regarding the article: The complete guide to OpenTelemetry in Golang

Please be civil.

[meobilivang]

Hi 👋 Thanks for this excellent article 🙇 I am a newbie to OTel, and your example is extremely easy to follow. I got the following problem and would to share my solution.

Description

I tried to run the TLS setup script but it didn't on my local environment:

• OS: Debian 11
• OpenSSL 1.1.1n

Got this error message: req: Unrecognized flag CA when trying to create and sign the certificates for Servers and Clients

Suggested Solution

• I looked through the man page for openssl for usage of req and x509 and some other examples of using the tool:

  • https://www.openssl.org/docs/man1.1.1/man1/openssl-req.html
  • https://www.openssl.org/docs/man1.1.1/man1/openssl-x509.html

• I found that we can break creating certificates and signing into 2 separate commands. The following commands work for me:

...
  { # create server certs.
    openssl \
      req \
      -new \
      -newkey rsa:2048 \
      -days 372 \
      -nodes \
      -keyout confs/server.key \
      -out confs/server.cst \
      -subj "/C=US/ST=CA/O=MyOrg/CN=myOrgCA" \
      -addext "subjectAltName=DNS:example.com,DNS:example.net,DNS:otel_collector,DNS:localhost" \

    # sign the cert
    openssl \
      x509 \
      -req \
      -in confs/server.cst -CA confs/rootCA.crt -CAkey confs/rootCA.key -CAcreateserial -out confs/server.crt
  }

  { # create client certs.
    openssl \
      req \
      -new \
      -newkey rsa:2048 \
      -days 372 \
      -nodes \
      -keyout confs/client.key \
      -out confs/client.cst \
      -subj "/C=US/ST=CA/O=MyOrg/CN=myOrgCA" \
      -addext "subjectAltName=DNS:example.com,DNS:example.net,DNS:otel_collector,DNS:localhost" \

    # sign the cert
    openssl \
      x509 \
      -req \
      -in confs/client.cst -CA confs/rootCA.crt -CAkey confs/rootCA.key -CAcreateserial -out confs/client.crt
  }
...

[xmarlem]

Thanks a lot for the excellent article!! thanks to you I have now a better idea about what to do. Thanks!

[komuw]

@meobilivang Hello,
Yeah the openssl script I had shared is not cross-platform. I had only shared what works for my machine and as you found out, it does not work across all OSes/openSSL versions. I'm glad to hear that you however found something that works for you.

[komuw]

Thanks a lot for the excellent article!! thanks to you I have now a better idea about what to do. Thanks!

@xmarlem Thanks and you are welcome.

[bnm3k]

This is the article that got me started in Telemetry in Golang earlier this year, so it's always a pleasure to re-read it 😊. Once more thank you.
I usually use zerolog but as you highlighted elsewhere, zerolog's hooks don't let us access the attributes when correlating with traces/spans. This shouldn't be a problem if log aggregators had some endpoint that lets us retrieve logs with a given TraceID/SpanID (aka plain-old SQL left joins). Alternatively, one could use zerolog's Logger.Output(w io.Writer) -> Logger to derive a child logger that tees the output to both the original destination (eg. std.Out) and to the Span (as an attribute)

[gcsfred2]

OS: macOS Ventura OpenSSL 3.1.2 1 Aug 2023 (Library: OpenSSL 3.1.2 1 Aug 2023)

#!/bin/bash
set -x # have bash print command been ran
set -e # fail if any command fails

setup_certs(){

  { # clean
    rm -rf confs/*.key
    rm -rf confs/*.crt
    rm -rf confs/*.cst
  }

  { # create CA.
    openssl \
      req \
      -new -verbose \
      -newkey rsa:4096 \
      -nodes \
      -x509 \
      -subj "/C=US/ST=CA/O=MyOrg/CN=myOrgCA" \
      -keyout confs/rootCA.key \
      -out confs/rootCA.crt
  }

  { # create server certs.

    openssl \
      req \
      -new -verbose \
      -subj "/C=US/ST=CA/O=MyOrg/CN=myOrgCA" -subject \
      -addext "subjectAltName = DNS:otel_collector,DNS:otel-col,DNS:otelcol,DNS:localhost,DNS:otlp,IP:127.0.0.1" \
      -addext "extendedKeyUsage = serverAuth" \
      -newkey rsa:2048 \
      -nodes -verify \
      -keyout confs/server.key \
      -out confs/server.cst \

    # sign the cert
    openssl \
      x509 \
      -req \
       -copy_extensions copy \
      -in confs/server.cst -CA confs/rootCA.crt -CAkey confs/rootCA.key -CAcreateserial -out confs/server.crt

  }

  {  # create client certs.
    openssl \
      req \
      -new -verbose \
      -newkey rsa:2048 \
      -nodes -verify \
      -keyout confs/client.key \
      -out confs/client.cst \
      -subj "/C=US/ST=CA/O=MyOrg/CN=myOrgCA" \
      -addext "subjectAltName = DNS:otel_collector,DNS:otel-col,DNS:otelcol,DNS:localhost,DNS:otlp,IP:127.0.0.1" \
      -addext "extendedKeyUsage = serverAuth"

    # sign the cert
    openssl \
      x509 \
      -req  \
       -copy_extensions copyall \
      -in confs/client.cst -CA confs/rootCA.crt -CAkey confs/rootCA.key -CAcreateserial -out confs/client.crt

  }

  { # clean
    rm -rf confs/*.csr
    rm -rf confs/*.srl

    chmod 666 confs/server.crt confs/server.key confs/rootCA.crt
  }

  { # verify 
     openssl x509 -in confs/server.crt -text -noout 
  }

}
setup_certs

I had 2 versions of openssl installed: one from conda another from homebrew. I used the one from homebrew. The -copy_extensions copyall argument was especially important. Without it, the certificates didn't have the

 X509v3 extensions:
            X509v3 Subject Alternative Name: 

portion.

It's very likely some DNS entries in subjectAltName above can be removed.

[komuw]

Hi @gcsfred2 , yes the openSSL commands are bound to have small differences across different OSes and/or across different openssl versions. The script included was meant to serve as an illustration.

[vamshiaruru32]

I just wanted to say, this was a great article. I haven't implemented in my own code yet, but I can't wait to do that.

[lorezi]

Thank you immensely for the insightful article! It was a game-changer for me, as I successfully implemented OpenTelemetry with Golang seamlessly for the first time. Your content made the process exceptionally clear and accessible. Grateful for your expertise!

[uyoaix]

@komuw Thank you very much for your article, it has been of great help to me.

About logs, I have another question that I would like to ask you. Now I want to create service observabilty with grafana , grafna loki , grafana tempo and prometheus . In my local machine, with golang service, I want to sent logs to loki with opentelemetry collector : app -> otel-collector -> loki I don't know if it can be implemented in this route. I search lots of articles, but I don't get the result.

In grafana website doc Send logs to Loki with Loki receiver , the route is: app/service -> logfile <- promtail -> otel-collector -> loki

I don't want to use promtail, because it will scrape container logs(not service log) when service deploy in docker or k8s.

[komuw]

@komuw Thank you very much for your article, it has been of great help to me.

About logs, I have another question that I would like to ask you. Now I want to create service observabilty with grafana , grafna loki , grafana tempo and prometheus . In my local machine, with golang service, I want to sent logs to loki with opentelemetry collector : app -> otel-collector -> loki I don't know if it can be implemented in this route. I search lots of articles, but I don't get the result.

In grafana website doc Send logs to Loki with Loki receiver , the route is: app/service -> logfile <- promtail -> otel-collector -> loki

I don't want to use promtail, because it will scrape container logs(not service log) when service deploy in docker or k8s.

Have you seen; https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/lokiexporter/README.md ? I haven't tried out loki, so I unfortunately do not have a lot of useful information to offer.

[uyoaix]

@komuw Thank you very much for your article, it has been of great help to me. About logs, I have another question that I would like to ask you. Now I want to create service observabilty with grafana , grafna loki , grafana tempo and prometheus . In my local machine, with golang service, I want to sent logs to loki with opentelemetry collector : app -> otel-collector -> loki I don't know if it can be implemented in this route. I search lots of articles, but I don't get the result. In grafana website doc Send logs to Loki with Loki receiver , the route is: app/service -> logfile <- promtail -> otel-collector -> loki I don't want to use promtail, because it will scrape container logs(not service log) when service deploy in docker or k8s.

Have you seen; https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/lokiexporter/README.md ? I haven't tried out loki, so I unfortunately do not have a lot of useful information to offer.

Okay, I got it. Thank you very much.

[vamshiaruru]

Just wanted to add here that global.SetMeterProvider(mp) doesn't seem to work anymore. You have to use otel.SetMeterProvider(mp)

[komuw]

Just wanted to add here that global.SetMeterProvider(mp) doesn't seem to work anymore. You have to use otel.SetMeterProvider(mp)

The main branch of https://github.com/komuw/otero usually has upto date code that is similar in shape to the blogpost.

[appiepollo14]

Hi,

Thanks for this very helpfull article. I've used it as a bases to implement OTEL in my service. I've a question about updating the loggers context when creating spans manually. For example within a service like this, I would expect to be creating more than one spans. What would the code look like when using zerolog? Do I need to update the logger with it's context each time, like in the code below. That feels like repeating dumb code. Any advice?

    ctx, span := otel.Tracer(tracerName).Start(r.Context(), "serviceB_HttpHandler")
    defer span.End()
    log := log.NewLogrus(ctx)
    log.Info("Logging from main span")

    ctx, subSpan := otel.Tracer(tracerName).Start(ctx, "subSpan")
    log = log.NewLogrus(ctx)
    log.Info("Logging from subSpan")
        subspan.End()

[komuw]

Hi,

Thanks for this very helpfull article. I've used it as a bases to implement OTEL in my service. I've a question about updating the loggers context when creating spans manually. For example within a service like this, I would expect to be creating more than one spans. What would the code look like when using zerolog? Do I need to update the logger with it's context each time, like in the code below. That feels like repeating dumb code. Any advice?

  ctx, span := otel.Tracer(tracerName).Start(r.Context(), "serviceB_HttpHandler")
  defer span.End()
  log := log.NewLogrus(ctx)
  log.Info("Logging from main span")

  ctx, subSpan := otel.Tracer(tracerName).Start(ctx, "subSpan")
  log = log.NewLogrus(ctx)
  log.Info("Logging from subSpan")
        subspan.End()

If you want the logs to have the items/details from the subspan, yes you would need to log.WithContext(subSpanCtx). I think the code you have provided works okay. I don't have any ideas at the moment on how to reduce the repetition.