search

Energinet-DataHub / opengeh-wholesale

Apache License 2.0
7 stars 3 forks source link

Prototype databricks health checks #1895

Closed mhj-halfspace closed 1 month ago

mhj-halfspace commented 9 months ago

STATUS 12/3-2024: Awaiting serverless. Currently placed first in fase 2. It might be planned in fase 1. Jesper will talk to Khatozen and update this issue.

Synopsis

As business and as a mandalorian I want to know if databricks is available and we have the necessary data available So we ensure it is running

Acceptance criteria

Description

We want to monitor if the databricks control plane is available, this will be done by calling SQL statement API endpoint to ensure that we can run a query through our workspace.

The query will access the data plane, and thereby ensure our databricks workspace has the correct access rights for the storage accounts. A simple query like select top(100) from metering points should be used.

BJM comments

We talked about selecting a solution that

  1. Adhered to the way we currently use health checks
  2. Supports monitoring these health checks the same way and in the same locations as current solution
  3. Avoid creating a new (cheap) cluster (as was earlier discussed), but rather use our current cluster

These "principles" led to the following conclusion:

Add the health check in the web app as it will support (1) and (2). Use our current cluster to implement the health check by doing a simple select of a few data. This ensures that the Databricks workspace has access to the data plane. We didn't see a solution regarding the job cluster that fits well with health checks as the cluster takes roughly 5 minutes to start each time. So it was considered out of scope.

mhj-halfspace commented 9 months ago

Blocked until serverless SQL is used.