argoproj / argo-workflows

Workflow Engine for Kubernetes
https://argo-workflows.readthedocs.io/
Apache License 2.0
15.11k stars 3.21k forks source link

test: simple DB CLI for local development #13715

Open MasonM opened 1 month ago

MasonM commented 1 month ago

Motivation

This adds a small CLI tool for developers to use when working with the DB locally. It provides two functions:

  1. Explicitly migrate the DB (which is normally only done when the workflow controller starts) to help test migrations:

    $ go run ./hack/db migrate
    INFO[0000] Migrating database schema                     clusterName=default dbType=postgres
    INFO[0000] applying database change                      change="CREATE INDEX argo_archived_workflows_i5 ON argo_archived_workflows (clustername, startedat)" changeSchemaVersion=60
  2. Insert randomly-generated archived workflows into the DB, which is intended to help test query performance (e.g. https://github.com/argoproj/argo-workflows/issues/13601)

    $ time go run ./hack/db/main.go fake-archived-workflows --rows 100000
    Using seed 2600415997737113470
    Clusters: [cx5j6rrzm5vnkfqqr4d]
    Namespaces: [bs98pxsfhs9z v95848 hq76xbj4kq7vrdp49vm ghzj6vcrth 262wb8w2b8v2wd2p2p9]
    Inserted 100000 rows
    
    real    17m56.554s
    user    1m53.833s
    sys     0m43.581s

    This is obviously not as efficient as it could be, but it's only intended to be run on an ad-hoc basis when manually testing query performance. To make it fast, we'd probably have to switch to direct SQL inserts, which would couple this script to the DB schema logic in persist/sqldb/workflow_archive.go.

Modifications

All logic is centralized in hack/db/main.go. I could've easily split that up, but that doesn't seem worth it, since this is only ever going to be used by developers.

Verification

Manually ran go run ./hack/db. See above for the output when using PostgreSQL. Here's the corresponding output for MySQL:

$ go run ./hack/db migrate --dsn mysql:password@tcp/argo
INFO[0000] Migrating database schema                     clusterName=default dbType=mysql

$ time go run ./hack/db/main.go fake-archived-workflows --rows 100 --dsn mysql:password@tcp/argo
Using seed 5504222375807483325
Clusters: [jckpg]
Namespaces: [7qr6rmgz5d8hz8zskt4 zst56t4wv9pkh8 jxlrvhtx4 g7bjn crk8qhxdtnddr2f]
Inserted 100 rows

real    0m3.261s
user    0m1.605s
sys     0m0.500s