Install Grafana in dev environment

[sevein]

Here's another small chunk of work to enable tracing. There are two changes:

1. I've installed Grafana after rendering the official chart template with helm template. With Tilt, you should be able to access Grafana using http://localhost:7490. Log in with username admin and password admin.
2. I've updated the API to emit traces automatically using the otelhttp package. This is just a starting point, later we could be adding spans from service methods as needed.

How to test:

1. Head to http://localhost:7490/explore
2. Use a TraceQL query like: {resource.service.name="enduro"} and hit Run query,
3. Choose a trace and explore it.

---

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 0% with 5 lines in your changes are missing coverage. Please review.

Project coverage is 47.21%. Comparing base (071e45a) to head (1b77cc3). Report is 2 commits behind head on main.

:exclamation: Current head 1b77cc3 differs from pull request most recent head 3dfca56. Consider uploading reports for the commit 3dfca56 to get more accurate results

Files	Patch %	Lines
internal/telemetry/traces.go	0.00%	3 Missing :warning:
internal/api/api.go	0.00%	1 Missing :warning:
main.go	0.00%	1 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #877 +/- ## ========================================== - Coverage 47.27% 47.21% -0.07% ========================================== Files 94 94 Lines 5087 5090 +3 ========================================== - Hits 2405 2403 -2 - Misses 2461 2466 +5 Partials 221 221 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[djjuhasz]

I don't know if it's intentional, but it looks like tracing is disabled for "enduro-internal" by default:

2024-03-08T18:45:20.251Z    V(0)    enduro  enduro/main.go:90   Starting... {"version": "0.1.0-t1b77cc376", "pid": 1}
2024-03-08T18:45:20.251Z    V(0)    enduro  enduro/main.go:93   Configuration file loaded.  {"path": "/home/enduro/.config/enduro.toml"}
2024-03-08T18:45:20.251Z    V(1)    enduro  enduro/main.go:98   Preservation config {"TaskQueue": "a3m"}
2024-03-08T18:45:20.251Z    V(1)    enduro  telemetry/traces.go:22  Tracing system is disabled. {"enabled": false, "addr": "", "sampling-ration": 1}
2024-03-08T18:45:20.386Z    V(0)    enduro.temporal-client  log/with_logger.go:74   Started Worker  {"level": "info", "Namespace": "default", "TaskQueue": "global", "WorkerID": "1@enduro-internal-546894d8df-5sc2s@"}
2024-03-08T18:46:01.957Z    V(1)    enduro  enduro/main.go:294  Starting new workflow   {"watcher": "dev-minio", "bucket": "sips", "key": "small.zip", "dir": false}
2024-03-08T18:46:01.971Z    V(2)    enduro.watcher  watcher/minio.go:105    Successfully removed message(s).    {"list": "minio-events-failed"}
2024-03-08T18:46:07.443Z    V(0)    enduro.temporal-client  log/with_logger.go:69   NewTimer    {"level": "debug", "Namespace": "default", "TaskQueue": "global", "WorkerID": "1@enduro-internal-546894d8df-5sc2s@", "WorkflowType": "storage-upload-workflow", "WorkflowID": "storage-upload-workflow-3e4ddab2-dfec-46cb-a792-18fa6d7137fe", "RunID": "9f2bbf32-222e-4673-9adf-c150e43dd6b8", "Attempt": 1, "TimerID": "5", "Duration": "15m0s"}
2024-03-08T18:46:07.562Z    V(0)    enduro.temporal-client  log/with_logger.go:69   ExecuteActivity {"level": "debug", "Namespace": "default", "TaskQueue": "global", "WorkerID": "1@enduro-internal-546894d8df-5sc2s@", "WorkflowType": "storage-move-workflow", "WorkflowID": "storage-move-workflow-3e4ddab2-dfec-46cb-a792-18fa6d7137fe", "RunID": "f7aebcf6-afaa-4888-9649-07154df8a512", "Attempt": 1, "ActivityID": "6", "ActivityType": "copy-to-permanent-location-activity"}

[djjuhasz]

I've run two transfers through Enduro with this branch:

But I'm not seeing any traces except a few "db-ping" traces :confused:

[djjuhasz]

Oh, I see now - only the API is producing traces

[sevein]

Glad that it's working. I hope it'll get a lot more interesting when start wrapping database connections, HTTP clients, Temporal, service boundaries, etc.