Observability

[nikita-volkov]

Adds a stream of events describing the significant state changes in the pool for monitoring purposes.

[nikita-volkov]

@robx Hey Rob! Can you please give a feedback on this?

[steve-chavez]

Many thanks for this @nikita-volkov! Just started experimenting on https://github.com/PostgREST/postgrest/pull/3229

Do you think hasql-pool could also expose metrics like "total waiters"? At one point (when hasql-pool used resource-pool) we had that one and other metrics, see https://github.com/PostgREST/postgrest/pull/2129.

[nikita-volkov]

Many thanks for this @nikita-volkov! Just started experimenting on PostgREST/postgrest#3229

Great! Thanks! Looking forward for feedback!

Do you think hasql-pool could also expose metrics like "total waiters"? At one point (when hasql-pool used resource-pool) we had that one and other metrics, see PostgREST/postgrest#2129.

Could you clarify on what you imply by "total waiters"? Unfortunately I couldn't derive that from the link.

[steve-chavez]

Could you clarify on what you imply by "total waiters"? Unfortunately I couldn't derive that from the link.

So the motivation is to be able to prevent poolAcquisitionTimeout, on the postgREST side this results in an API error (ref).

For this, I understand we could use the following metrics:

1) pending_requests (same idea as total waiters) https://opentelemetry.io/docs/specs/semconv/database/database-metrics/#metric-dbclientconnectionspending_requests

• If this keeps increasing it would be an indicative that a pool acquisition timeout will happen. So an admin can adjust the pool size or tune the queries to prevent it.
  • I've also seen this named as "concurrentWaiters: Number of waiting threads" (ref)

2) pool wait time https://opentelemetry.io/docs/specs/semconv/database/database-metrics/#metric-dbclientconnectionswait_time

• Same idea as above, if it keeps increasing then a pool timeout will happen.

[nikita-volkov]

Thanks for the references! I'll see what can be done.

[nikita-volkov]

I've taken a look. Seems like both of those metrics are already computable without the features involved in this PR.

executeObservedSession :: Pool -> Monitor -> Session -> IO ()
executeObservedSession pool monitor session = do
  Monitor.incGauge monitor "db.client.connections.pending_requests"
  startTime <- getCurrentTime
  Pool.use pool do
    liftIO $ do
      endTime <- getCurrentTime
      Monitor.observeHistogram monitor "db.client.connections.wait_time" (diffUTCTime endTime startTime)
      Monitor.decGauge monitor "db.client.connections.pending_requests"
    session

@steve-chavez Is that correct?

[steve-chavez]

@nikita-volkov You're right! Thanks for the snippet!

[steve-chavez]

Monitor.observeHistogram monitor "db.client.connections.wait_time" (diffUTCTime endTime startTime)

Curious, where does observeHistogram come from? I found decGauge on hoogle but not observeHistogram.

[nikita-volkov]

Ah. It was just pseudocode. I was implying something like this.

[nikita-volkov]

Thanks for cooperation guys! It's released.