search

open-telemetry / opentelemetry-demo

This repository contains the OpenTelemetry Astronomy Shop, a microservice-based distributed system intended to illustrate the implementation of OpenTelemetry in a near real-world environment.
https://opentelemetry.io/docs/demo/
Apache License 2.0
1.76k stars 1.12k forks source link

Observability during deployment #359

Open reyang opened 2 years ago

reyang commented 2 years ago

Feature Request

The current demo seems to focus on observability when everything is already up and running. I feel it might be important (and also cool) to be able to observe the system during setup, deployment or even decommission.

For example, maybe during the docker compose up, the 1st service to start is actually the UI (maybe Grafana or Kibana), which we can use to easily tell:

  1. Which service is now up and running.
  2. Any service is under deployment.
  3. Any deployment issue.

Describe the solution you'd like:

Maybe:

  1. Make sure the logging/metrics backend/UI are up and running.
  2. Deploy and run other services.
  3. Send deployment logs and service up/availability metrics during the deployment.
wph95 commented 1 year ago

A possible scenario, using our helm-demo as an example

STEP

  1. helm install otel-demo --step= bootstrap Install only our frontUI When you visit our frontUI, you can see that the other services are offline.

  2. helm install otel-demo --step=monitor

    Install prometheus/grafana

  3. helm install otel-demo --step=demo-app

    install our otel-demo While waiting, you can see the installation process through the frontUI admin page / grafana

Parts to discuss

  1. step1 and step2 may be combined into one step.

  2. How to gracefully turn on trace and log is something we can implement later.

  3. It means that our helm chart has two modes.

    1. easy mode, install all services with one click
    2. step by step mode, you can enjoy Observability during deployment