search

argoproj / argo-workflows

Workflow Engine for Kubernetes
https://argo-workflows.readthedocs.io/
Apache License 2.0
14.55k stars 3.12k forks source link

argo submit slow when calling multiple templates. suggest improving caching #7418

Open rwong2888 opened 2 years ago

rwong2888 commented 2 years ago

I'm trying argo -n argo submit --from clusterworkflowtemplate/<template> -v --gloglevel=9 ... and it is taking about 10-12 seconds to complete. Looks like it spends most time resolving all the templates...

Submit from UI is also slow https://slack-files.com/T08PSQ7BQ-F02QV220QM8-f33032ca57

Logs https://cloud-native.slack.com/archives/C01QW9QSSSK/p1639579998354400?thread_ts=1639420502.300600&cid=C01QW9QSSSK

Slack Archive for reference https://cloud-native.slack.com/archives/C01QW9QSSSK/p1639420502300600

crenshaw-dev commented 2 years ago

A couple things I noticed from the logs: 1) Requests are being throttled. Maybe tweaking the --qps and --burst flags on the workflow-controller could help? 2) Templates are being fetched multiple times. Is there a bug in the caching mechanism / is any caching happening?

terrytangyuan commented 2 years ago

Which version are you using? Are you also seeing this issue in older versions?

rwong2888 commented 2 years ago

@terrytangyuan v3.2.4