search

crosscloudci / cross-project

App deployment for CNCF CI Project
https://cncf.ci
5 stars 5 forks source link

helm charts repo errors - Error: failed to download (hint: running `helm repo update` may help) #27

Closed agentpoyo closed 3 years ago

agentpoyo commented 3 years ago

Description

Error: failed to download (hint: running helm repo update may help)

Long Description:

We are seeing the effects of the helm chart repo's being deprecated during maintenance mode from helm. projects slated to push to Equinix Metal clusters are seeing the following errors when attempting to install the latest stable charts:

$ echo Removing any previous deploys of "${DEPLOYMENT_NAME}"
Removing any previous deploys of fluentd-master
$ helm list --filter "${DEPLOYMENT_NAME}" > /dev/null && helm delete "${DEPLOYMENT_NAME}" || true
Error: uninstall: Release not loaded: fluentd-master: release: not found
$ helm repo add stable https://kubernetes-charts.storage.googleapis.com/
"stable" has been added to your repositories
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈ Happy Helming!⎈ 
$ if [ "$TARGET_PROJECT_NAME" == so ]; then # collapsed multi-line command
skipping onap configuration
$ if [ "$TARGET_PROJECT_NAME" == prometheus ]; then # collapsed multi-line command
$ helm install --namespace "${NAMESPACE}" "${DEPLOYMENT_NAME}" $EXTRA_ARGS $IMAGE_ARGS $TAG_ARGS "${CHART_REPO}"/"${CHART}" --wait --timeout 60m
Error: failed to download "stable/fluentd" (hint: running `helm repo update` may help)
ERROR: Job failed: exit code 1

Suggested Fix:

Update to use our cncf.gitlab.io/stable helm repo

agentpoyo commented 3 years ago

Discovered that the helm chart repo's that were cloned were all updated as "deprecated = true" so this caused failures for helm deploys on my initial trigger build test.

agentpoyo commented 3 years ago

Each project is seeing different errors when attempting to use our own cncf.gitlab.io/stable repo URL:

fluentd:

$ kubectl get componentstatuses
NAME                 STATUS      MESSAGE                                                                                       ERROR
scheduler            Unhealthy   Get "http://127.0.0.1:10251/healthz": dial tcp 127.0.0.1:10251: connect: connection refused   
controller-manager   Unhealthy   Get "http://127.0.0.1:10252/healthz": dial tcp 127.0.0.1:10252: connect: connection refused   
etcd-0               Healthy     {"health":"true"}                                                                             
$ env | grep IMAGE
CI_REGISTRY_IMAGE=registry.cidev.cncf.ci/cncf/cross-project
FF_DOCKER_HELPER_IMAGE_V2=false
IMAGE_ARGS=--set image.repository=registry.cidev.cncf.ci/fluent/fluentd
$ env | grep TAG
CI_RUNNER_TAGS=amd64
CI_BUILD_STAGE=Cross-Project
TAG_ARGS=--set image.tag=v1-11-5.47814.24fe4cbc.amd64
CI_JOB_STAGE=Cross-Project
$ echo Removing any previous deploys of "${DEPLOYMENT_NAME}"
Removing any previous deploys of fluentd-v1115
$ helm list --filter "${DEPLOYMENT_NAME}" > /dev/null && helm delete "${DEPLOYMENT_NAME}" || true
Error: uninstall: Release not loaded: fluentd-v1115: release: not found
$ helm repo add stable http://cncf.gitlab.io/stable
"stable" has been added to your repositories
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈ Happy Helming!⎈ 
$ if [ "$TARGET_PROJECT_NAME" == so ]; then # collapsed multi-line command
skipping onap configuration
$ if [ "$TARGET_PROJECT_NAME" == prometheus ]; then # collapsed multi-line command
$ helm install --namespace "${NAMESPACE}" "${DEPLOYMENT_NAME}" $EXTRA_ARGS $IMAGE_ARGS $TAG_ARGS "${CHART_REPO}"/"${CHART}" --wait --timeout 60m
coalesce.go:160: warning: skipped value for image: Not a table.
Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "DaemonSet" in version "extensions/v1beta1"
ERROR: Job failed: exit code 1

prometheus:

$ env | grep IMAGE
CI_REGISTRY_IMAGE=registry.cidev.cncf.ci/cncf/cross-project
FF_DOCKER_HELPER_IMAGE_V2=false
IMAGE_ARGS=--set server.image.repository=registry.cidev.cncf.ci/prometheus/prometheus
$ env | grep TAG
CI_RUNNER_TAGS=amd64
CI_BUILD_STAGE=Cross-Project
TAG_ARGS=--set server.image.tag=v2.22.1.00f16d1a.196223.amd64
CI_JOB_STAGE=Cross-Project
$ echo Removing any previous deploys of "${DEPLOYMENT_NAME}"
Removing any previous deploys of prometheus-v2221
$ helm list --filter "${DEPLOYMENT_NAME}" > /dev/null && helm delete "${DEPLOYMENT_NAME}" || true
Error: uninstall: Release not loaded: prometheus-v2221: release: not found
$ helm repo add stable http://cncf.gitlab.io/stable
"stable" has been added to your repositories
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈ Happy Helming!⎈ 
$ if [ "$TARGET_PROJECT_NAME" == so ]; then # collapsed multi-line command
skipping onap configuration
$ if [ "$TARGET_PROJECT_NAME" == prometheus ]; then # collapsed multi-line command
$ helm install --namespace "${NAMESPACE}" "${DEPLOYMENT_NAME}" $EXTRA_ARGS $IMAGE_ARGS $TAG_ARGS "${CHART_REPO}"/"${CHART}" --wait --timeout 60m
Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "Deployment" in version "extensions/v1beta1"
ERROR: Job failed: exit code 1

envoy:

$ env | grep IMAGE
CI_REGISTRY_IMAGE=registry.cidev.cncf.ci/cncf/cross-project
FF_DOCKER_HELPER_IMAGE_V2=false
IMAGE_ARGS=--set image.repository=registry.cidev.cncf.ci/envoyproxy/envoy
$ env | grep TAG
CI_RUNNER_TAGS=amd64
CI_BUILD_STAGE=Cross-Project
TAG_ARGS=--set image.tag=v1.15.2.c2a3cbd3.196255.amd64
CI_JOB_STAGE=Cross-Project
$ echo Removing any previous deploys of "${DEPLOYMENT_NAME}"
Removing any previous deploys of envoy-v1152
$ helm list --filter "${DEPLOYMENT_NAME}" > /dev/null && helm delete "${DEPLOYMENT_NAME}" || true
Error: uninstall: Release not loaded: envoy-v1152: release: not found
$ helm repo add stable http://cncf.gitlab.io/stable
"stable" has been added to your repositories
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈ Happy Helming!⎈ 
$ if [ "$TARGET_PROJECT_NAME" == so ]; then # collapsed multi-line command
skipping onap configuration
$ if [ "$TARGET_PROJECT_NAME" == prometheus ]; then # collapsed multi-line command
$ helm install --namespace "${NAMESPACE}" "${DEPLOYMENT_NAME}" $EXTRA_ARGS $IMAGE_ARGS $TAG_ARGS "${CHART_REPO}"/"${CHART}" --wait --timeout 60m
Error: rendered manifests contain a resource that already exists. Unable to continue with install: Service "envoy-stable" in namespace "default" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-name" must equal "envoy-v1152": current value is "envoy-v1150"
ERROR: Job failed: exit code 1
agentpoyo commented 3 years ago

Lower environments fixed; rebased all branches as they were too far ahead with commits; rebase production which should reflect changes tonight for metal deploys.

Seems envoy still potentially fails as it's trying to pass a command to start that's been deprecated in newer versions but the charts are behind.

agentpoyo commented 3 years ago

Equinix Metal deploys are now working using our new helm repo at cncf.gitlab.io