timescale / tobs

tobs - The Observability Stack for Kubernetes. Easy install of a full observability stack into a k8s cluster with Helm charts.
Apache License 2.0
563 stars 59 forks source link

Remove code related to tobs CLI #471

Closed onprem closed 2 years ago

onprem commented 2 years ago

Description

Fixes #396

Type of change

What type of changes does your code introduce to tobs? Put an x in the box that apply.

paulfantom commented 2 years ago

It seems like the issue might be with github actions not having enough resources to run the whole stack:

Warning FailedScheduling 6s (x5 over 67s) default-scheduler 0/1 nodes are available: 1 Insufficient cpu. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.

Could you test this theory by temporarily removing resource requests from prometheus, promscale, and timescaledb pods (done in chart/values.yaml)

If that is the case, we can create values.yaml patches and place them in chart/ci/XYZ-values.yaml files as per ct install docs:

Charts may have multiple custom values files matching the glob pattern '*-values.yaml' in a directory named 'ci' in the root of the chart's directory. The chart is installed and tested for each of these files. If no custom values file is present, the chart is installed and tested with defaults.

onprem commented 2 years ago

Waiting on #493 as that will fix the timeout issue with ct install.

onprem commented 2 years ago

It seems like the issue might be with github actions not having enough resources to run the whole stack:

Warning FailedScheduling 6s (x5 over 67s) default-scheduler 0/1 nodes are available: 1 Insufficient cpu. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.

Could you test this theory by temporarily removing resource requests from prometheus, promscale, and timescaledb pods (done in chart/values.yaml)

If that is the case, we can create values.yaml patches and place them in chart/ci/XYZ-values.yaml files as per ct install docs:

Charts may have multiple custom values files matching the glob pattern '*-values.yaml' in a directory named 'ci' in the root of the chart's directory. The chart is installed and tested for each of these files. If no custom values file is present, the chart is installed and tested with defaults.

Looks like the case to me as well. We already have a values.yaml there in this PR to disable telemetry. I'll remove the resource requests and then test.

onprem commented 2 years ago

Now the CI is failing because ct deletes the namespace in the end, with a hardcoded timeout of 180s. But with PVCs and finalisers, tobs almost always takes longer than 180s to delete the namespace, causing ct install to fail.

Upstream issue: helm/chart-testing#227

onprem commented 2 years ago

The workaround the namespace deletion timeout issue is to do ct install in an existing namespace. This way, ct doesn't try to remove the namespace, bypassing the namespace uninstall timeout.

There is a new problem though, ct runs helm test after an install, and since all helm commands have common extraArgs in configuration, and we need --wait for install, helm test fails as it doesn't have a flag called --wait.

image

Update: since the --wait flag is implicitly added by ct here, I just removed the flag, getting helm test to work.