Open brien-gleason opened 1 month ago
I'm almost sure the main reason of this behavior terribly poor performance of filtering runs by tags. Your repro above confirms it. https://github.com/dagster-io/dagster/blob/1.7.13/python_modules/dagster/dagster/_core/storage/runs/sql_run_storage.py#L344-L360
This is generic approach, for sure, but with current database design performance for this particular query will slow down as the number of runs/runs tags increases. I guess you faced this because your database accumulated a lot of runs and runs tags as well. We got the same behavior on 1.6.8 with MySQL for schedules/sensors pages. Currently we have around 100k runs and 700k runs tags.
FYI: relational databases always have an issues with multiple filtering conditions combined by OR clause.
Dagster version
1.7.11
What's the issue?
When I navigate to the Runs view or Schedules view of a specific job, I get an error and see
GraphQL Error - see console for details
.Both pages return this error:
Message: (psycopg2.errors.QueryCanceled) canceling statement due to statement timeout
. The full stacktraces are in "Additional Information" section.I don't see this error on previous versions and we did not change anything on our Postgres DB. Rolling back to version 1.7.10 fixes these two pages. The queries both complete successfully when running directly on the DB.
How to reproduce?
This happens for me every time I navigate to the Runs or Schedules views on version 1.7.11 or 1.7.12.
Deployment type
Dagster Helm chart
Deployment details
Kubernetes version is 1.28.5 and Postgres DB is an Azure Flex DB running on version 14.11
Additional information
The full Stacktrace for the Schedules page is:
The full stacktrace for Runs page is:
Message from the maintainers
Impacted by this issue? Give it a 👍! We factor engagement into prioritization.