Prototype databricks health checks

STATUS 12/3-2024: Awaiting serverless. Currently placed first in fase 2. It might be planned in fase 1. Jesper will talk to Khatozen and update this issue.

Synopsis

As business and as a mandalorian I want to know if databricks is available and we have the necessary data available So we ensure it is running

Acceptance criteria

[ ] Created call from .NET api to databricks query api with simple query
[ ] Send result to app service health check

Description

We want to monitor if the databricks control plane is available, this will be done by calling SQL statement API endpoint to ensure that we can run a query through our workspace.

The query will access the data plane, and thereby ensure our databricks workspace has the correct access rights for the storage accounts. A simple query like select top(100) from metering points should be used.

BJM comments

We talked about selecting a solution that

Adhered to the way we currently use health checks
Supports monitoring these health checks the same way and in the same locations as current solution
Avoid creating a new (cheap) cluster (as was earlier discussed), but rather use our current cluster

These "principles" led to the following conclusion:

Add the health check in the web app as it will support (1) and (2). Use our current cluster to implement the health check by doing a simple select of a few data. This ensures that the Databricks workspace has access to the data plane. We didn't see a solution regarding the job cluster that fits well with health checks as the cluster takes roughly 5 minutes to start each time. So it was considered out of scope.

Energinet-DataHub / opengeh-wholesale