Open nschmeller opened 1 year ago
I think you're misreading the docs.
local_resource readiness checks apply to the serve_cmd, not the build cmd. The helm_repo function uses the build cmd. I didn't see anything in your logs to suggest grafana deployed before grafana-charts finished.
Gotcha, thanks for the clarification -- could there be another reason why this would be flaky / racy?
You say you expected it to wait on resource_deps, but I don't see any resource_deps in the code snippet you posted?
My bad, I pasted the wrong snippet from the Tiltfile 😅 -- the failed CI run happened after I committed code with the resource_deps
added:
diff --git a/monitoring/Tiltfile b/monitoring/Tiltfile
index 30cb8f76e4..e0a3696cd7 100644
--- a/monitoring/Tiltfile
+++ b/monitoring/Tiltfile
@@ -56,6 +56,7 @@ helm_repo(name='grafana-charts',
helm_resource(name='grafana',
chart='grafana-charts/grafana',
flags=['--values', './dev/grafana/values.yaml'],
+ resource_deps=['grafana-charts'],
port_forwards=[3000])
watch_file('./dev/grafana/values.yaml')
So the state of the Tiltfile when this happened was actually
# Install Grafana Helm Charts
helm_repo(name='grafana-charts',
url='https://grafana.github.io/helm-charts')
helm_resource(name='grafana',
chart='grafana-charts/grafana',
flags=['--values', './dev/grafana/values.yaml'],
resource_deps=['grafana-charts'],
port_forwards=[3000])
watch_file('./dev/grafana/values.yaml')
Sorry again for the confusion -- any ideas still?
I'm not 100% confident 'helm repos add' is concurrency safe. From your logs, it looks like two repo-adds (repos-add?) ran at the same time, but only one printed, which is a bit suspicious. Maybe we could try removing the allow_parallel param from helm_repo and see if that helps?
I'm not sure how to force the race, but I added the problematic code to the test Tiltfile in helm_resource
in the hopes of reproducing. Let me know how that PR looks!
Hi all, I bumped into a race condition recently in a CI run where
helm_resource
was supposed to wait forhelm_repo
to finish but did not. Here are some logs from what happened:The Tiltfile section that caused that is
Here,
grafana
has a dependency ongrafana-charts
, but in the logs above, the chart was not found!I went ahead and did some triaging in the hopes that it would be useful. My suspicion for what happened:
helm_repo
uses the Tiltfilelocal_resource
API (https://github.com/tilt-dev/tilt-extensions/blob/13280048812f95896150b6fcf23462bb501d250d/helm_resource/Tiltfile#L158)local_resource
returns "ready" to Tilt only when the process is started (and not when it is finished!), ctx: https://docs.tilt.dev/local_resource.html#readiness_probe.helm_repo
is called and starts thehelm repo add
helm_resource
, which is dutifully waiting onhelm_repo
to signal "ready", gets the all-clear from Tilt as the process was startedhelm_resource
attempts to install the charthelm_resource
crashes when it does not find the charthelm_repo
finishes installing the chart, alas too lateI propose that
helm_repo
's call tolocal_resource
adds a newreadiness_probe
which uses anexec_action
ofhelm repo list -o json | jq -e 'map(select(.name == <name-of-chart>)) != []'
to determine success of chart installation.Let me know if this all makes sense please!