Metrics Calculation Service: Rewrite protoype arrow flight server to new service

[ravenac95]

What is it?

The metrics calculation service is responsible for managing our duckdb backed cluster for metrics calculation. This makes it so that the sqlmesh code doesn’t directly instantiate the cluster from the multiple threads of execution

Notes

• For now we will do things with polling for asynchronous things as there aren’t many consumers of the service

Endpoints

• GET /status
  • Simple liveness endpoint
  • Params
    • N/A
  • Returns
    • 200
      • This will mean the service has started
• POST /cluster/start
  • Idempotent request to ensure the cluster is started.
  • Params
    • min_size - cluster minimum size
    • max_size - cluster maximum size
  • Returns
    • 200
• GET /cluster/status
  • Gets the current status
  • Params
    • N/A
  • Returns
    • 200
      • isReady: bool - True if the cluster is ready to operate
• POST /job/submit
  • Submits query job for calculation
  • Params
    • Query to execute
    • Time range
    • Dependent tables
    • Dependent table mappings
    • GCS path to write to
  • Returns
    • Job-id
• POST /job/status
  • Returns the status of a metrics calculation job