argoproj / argo-workflows

Workflow Engine for Kubernetes
https://argo-workflows.readthedocs.io/
Apache License 2.0
15.11k stars 3.21k forks source link

fix: remove JSON cast when querying archived workflows #13777

Closed MasonM closed 1 month ago

MasonM commented 1 month ago

Motivation

With PostgreSQL, the argo_archived_workflows.workflow column has been of type json ever since 8a1e611a03da8374567c9654f8baf29b66c83c6e, which was released as v2.5.0. Therefore, the ::json casts do nothing, and prevent users from improving performance by migrating to JSONB using the following query:

alter table argo_archived_workflows alter column workflow set data type jsonb using workflow::jsonb

Without the changes in this PR, running the above will massively slow down the queries, because casting JSONB to JSON is expensive. With the changes, I'm measuring a ~80% performance boost (see below). Other users have reported similar improvements.

The only downside to migrating to JSONB is it can take a long time if you've got a ton of workflows (~72s on my local DB with 100,000 workflows). I entered https://github.com/argoproj/argo-workflows/pull/13779 with the migration, but I'm entering this separately so it can hopefully go out in 3.6.0. That way, we can tell users to run the above query if they run into performance issues, without having to do a patch release with the migration.

Modifications

Replace (workflow::json) with just workflow

Verification

See https://github.com/argoproj/argo-workflows/pull/13779

MasonM commented 1 month ago

Here's the PR for the migration, which builds on top of this PR: https://github.com/argoproj/argo-workflows/pull/13779

agilgur5 commented 3 weeks ago

@jiachengxu Would you mind reviewing this?

@terrytangyuan Jiacheng is also apparently not in the "members" team a la https://github.com/argoproj/argo-workflows/issues/8790#issuecomment-2412016304, so that seems to be why we can't directly "request a review" via GH

terrytangyuan commented 3 weeks ago

Added @jiachengxu to members

agilgur5 commented 3 weeks ago

Thanks Terry 👍 EDIT: also can confirm review requests can be done now as with https://github.com/argoproj/argo-workflows/pull/13819#event-14892805258