OpenTelemetry & Parser/Eval refactoring

[dobarx]

Improvements

OpenTelemetry support

• 82

Exporting telemetry

Global --debug flag

Added global CLI --debug flag to save logs, traces and metrics to disk. This works for all commands.

   $ fabric render document.example --debug
   ...
   $ tree .fabric/
   .fabric
   └── debug
       └── 240512185714
           ├── logs.json
           ├── metrics.json
           └── traces.json

Exported files can be imported to other tools for inspection. They are in standard OpenTelemetry json format.

OpenTelemetry Protocol

When using fabric CLI, OpenTelemetry protocol can be enabled using environment variables. These are the default settings:

   FABRIC_OTELP_ENABLED=false
   FABRIC_OTELP_URL=https://otelp.blackstork.io

To test this, you can setup a local OpenTelemetry-Collector using docker-compose. You can use this repository: otel-compose. This will setup Collector, Loki, Tempo, Prometheus & Grafana. Next, configure environment variables:

   export FABRIC_OTELP_ENABLED=true
   export FABRIC_OTELP_URL=http://localhost:4318

Now all logs, traces & metrics are visible on Grafana:

   http://localhost:3000/explore

Example of logql query for Loki:

   {job="fabric"} | json

Using as Go library

If fabric is used as Go library, then a separate logger & tracer can be provided to configure telemetry exports. It can be sent to the collector or even stored somewhere on the database and shown to the user in the web dashboard. This does not use global state and won't trigger other loggers & tracers. Defaults are noop-Logger & noop-Tracer.

import "github.com/blackstork-io/fabric/engine"
...
eng := engine.New(
   engine.WithLogger(logger),
   engine.WithTracer(tracer),
)
defer eng.Cleanup()
diag := eng.ParseDir(ctx, fs)
...

Secret specs

Now logger logs also plugin config & arguments. To hide secrets new property was added to dataspec named Secret. This allows to replace plugin config & args values with <secret>. Example:

&dataspec.AttrSpec{
    Name:        "password",
    Type:        cty.String,
    Constraints: constraint.RequiredNonNull,
    Secret:      true,
}

This will be logged as: password=<secret>.

We could also use this property in the future to minimize risk of people leaving their secrets in .hcl files. There are two options:

• Throw an error if secret is not provided using from_env_variable().
• Show warnings about secrets in .hcl templates that do not use from_env_variable().

Parser/Eval refactoring

To improve logging & add tracing some refactoring was done on parser & eval logic. This also makes it easier to use fabric as Go library.

• /parser - removed plugin caller and moved out most of evaluation logic to seperate /eval package. /parser package now only parses hcl and does not execute plugins.
• /eval package added for executing parsed blocks.
• /cmd.Evaluator moved to separate /engine package that can also be used as Go library. It does not use any global state.
• /test/e2e moved to /engine package because these tests were basically only testing cmd.Evaluator.
• /pkg/fabctx moved to /cmd/internal/fabctx since it's only used by /cmd package for graceful shutdowns.

Other stuff

• Replaced most of func return signatures with diagnostics.Diag where we still used hcl.Diagnostics. Makes it easier to handle diagnostics and less jumping between Go types.
• Version command fix. fabric --version did not work when fabric is built using go install because it sets version to rootCmd before init() updates version variable with the one extracted from build info. This resulted with rootCmd.Version == "".
• Recovery from panics to persist telemetry data.

Future

• We could also add OpenTelemetry to plugin registry API and propagate tracing spans IDs via http headers using otel sdk. This would make imported traces & logs from fabric CLI to have strong correlation with our API traces & logs. This would help to debug issues with commands like install that calls our API. We would be able to inspect traces of CLI & API in same place.