search

mlflow / mlflow-go

Repository for the Go-based MLflow Tracking Server
Apache License 2.0
5 stars 4 forks source link

Deadlock with large amount of metrics LogBatch #81

Open nojaf opened 3 weeks ago

nojaf commented 3 weeks ago

While running some k6 benchmarks with logBatch (of 500 metrics), I noticed this error:

ERRO[0516] Error encountered in POST /api/2.0/mlflow/runs/log-batch: [INTERNAL_ERROR] error updating latest metrics for run_uuid "3c0269b4a45448b7b21d6f2a9c5ae2b1": failed to upsert latest metrics for run_uuid "3c0269b4a45448b7b21d6f2a9c5ae2b1": ERROR: deadlock detected (SQLSTATE 40P01)

We will need to investigate why there is a deadlock here.