Closed pthornton closed 1 year ago
ApplicationSet that creates application for above chart:
---
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: react-frontend-service-test
labels:
ring: test
spec:
generators:
- git:
repoURL: https://gitlab.com/somewhere/prt/example-apps/mushu-dev.git
revision: HEAD
files:
- path: "config/**/config.json"
template:
metadata:
name: 'react-frontend-service-{{cluster.name}}'
labels:
ring: test
tenant: '{{cluster.name}}'
domain: '{{cluster.domain}}'
spec:
project: '{{argocd.project}}'
source:
repoURL: https://gitlab.com/somwhere/prt/example-apps/mushu-dev.git
targetRevision: HEAD
path: "./react-frontend-service/helm_base"
helm:
version: v3
releaseName: 'react-frontend-service-{{cluster.name}}'
valueFiles:
- ../overlays/{{cluster.name}}/globals.yaml
- ../overlays/flags.yaml
- ../overlays/{{cluster.name}}/values.yaml
- ../overlays/{{cluster.name}}/hotfix.yaml
destination:
server: '{{cluster.address}}'
namespace: 'example-test'
ignoreDifferences:
- group: apps
jsonPointers:
- /spec/replicas
kind: Deployment
namespace: 'example-test'
# Sync policy
syncPolicy:
automated: # automated sync by default retries failed attempts 5 times with following delays between attempts ( 5s, 10s, 20s, 40s, 80s ); retry controlled using `retry` field.
prune: true # Specifies if resources should be pruned during auto-syncing ( false by default ).
selfHeal: true # Specifies if partial app sync should be executed when resources are changed only in target Kubernetes cluster and no git change detected ( false by default ).
allowEmpty: false # Allows deleting all application resources during automatic syncing ( false by default ).
syncOptions: # Sync options which modifies sync behavior
- CreateNamespace=true # Namespace Auto-Creation ensures that namespace specified as the application destination exists in the destination cluster.
- Validate=true # disables resource validation (equivalent to 'kubectl apply --validate=false') ( true by default ).
Worked in v2.0.5 helm chart 3.12.1. Starts failing in v2.1.0 helm chart 3.17.1.
@AustinSmart t I noticed you had some comments in other threads around OCI chart deps being broken in 2.1.0.... Is this related?
Since there is a oci chart as a dependency in the first comment, yes it’s likely related. I don’t think the oci chart dependency bug has been fixed yet.
Is there a workaround for that @AustinSmart ? Couldn't find any reference as to exactly why that functionality broke.
time="2022-01-19T16:45:42Z" level=error msg="finished unary call with code Unknown" error="Manifest generation error (cached): `helm repo add 712053168757.dkr.ecr.us-west-2.amazonaws.com oci://712053168757.dkr.ecr.us-west-2.amazonaws.com/helm/high-five-templates-typescript-backend-service` failed exit status 1: Error: looks like \"oci://712053168757.dkr.ecr.us-west-2.amazonaws.com/helm/high-five-templates-typescript-backend-service\" is not a valid chart repository or cannot be reached: tag explicitly required" grpc.code=Unknown grpc.method=GenerateManifest grpc.request.deadline="2022-01-19T16:50:42Z" grpc.service=repository.RepoServerService grpc.start_time="2022-01-19T16:45:42Z" grpc.time_ms=1.839 span.kind=server system=grpc
Might be related to 7104
Support for the oci:// protocol is also available in various other subcommands. Here is a complete list:
helm pull
helm show
helm template
helm install
helm upgrade
Note that "helm repo" is missing....
I had the same issue (also AWS ECR repo that connected well but the same 401 error when adding an application).
The regression probably came in with (lines 195 to 197 in RepoAdd()
in util/helm/cmd.go
on October 30th 2021):
if c.helmPassCredentialsSupported && passCredentials {
args = append(args, "--pass-credentials")
}
This is not what we need to login to the ECR, we need it without the --pass-credentials flag for helm. So for instance, this works to login:
aws ecr get-login-password --region us-east-1 | helm registry login --password-stdin --username AWS "$(aws sts get-caller-identity --query Account --output text).dkr.ecr.us-east-1.amazonaws.com"
I haven't dug into why this is so, but I have a temporary workaround that is working for me with v2.2.3. You can simply set passCredentials
to false in your Application
definition:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: xxxx
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: xxxx
source:
repoURL: xxxx
path: xxxx
targetRevision: xxxx
helm:
passCredentials: false
...
@mfbieber I'm setting this in the application but I'm not sure it's getting honored. I'm using ApplicationSet to create the application. The ApplicationSet shows:
Source:
Helm:
Pass Credentials: false
Release Name: typescript-backend-service-{{cluster.name}}
Value Files:
../overlays/{{cluster.name}}/globals.yaml
../overlays/flags.yaml
../overlays/{{cluster.name}}/values.yaml
../overlays/{{cluster.name}}/hotfix.yaml
The created Application has no mention of passCredentials:
Source:
Helm:
Release Name: typescript-backend-service-a0000000027
Value Files:
../overlays/a0000000027/globals.yaml
../overlays/flags.yaml
../overlays/a0000000027/values.yaml
../overlays/a0000000027/hotfix.yaml
Know of any way to verify passCredentials value on the Application? I see a --helm-pass-credentials on argocd cli set command, but nothing on get? Both get/set fail with no permission to access oci. Starting to think the ApplicationSet is not honoring the setting.
@mfbieber What version are you on? I created the application directly and same failure on 3.29.5/2.2.2.
I created the Application directly and it still fails with same error:
Source:
Helm:
Pass Credentials: false
Release Name: typescript-backend-service-a0000000027
Value Files:
../overlays/a0000000027/globals.yaml
../overlays/flags.yaml
../overlays/a0000000027/values.yaml
../overlays/a0000000027/hotfix.yaml
Are you calling this directly yourself?
aws ecr get-login-password --region us-east-1 | helm registry login --password-stdin --username AWS "$(aws sts get-caller-identity --query Account --output text).dkr.ecr.us-east-1.amazonaws.com"
Have a workaround using plugin. In values.yaml:
repoServer:
volumes:
- name: custom-tools
emptyDir: {}
- name: plugin-scripts
configMap:
name: plugin-scripts
defaultMode: 0755
initContainers:
- name: helm-setup
image: alpine:3
command: [
"/bin/sh",
"-c",
"apk add --no-cache aws-cli;
which aws;
aws --version;
cp /usr/bin/aws /custom-tools/aws; "]
volumeMounts:
- mountPath: /custom-tools
name: custom-tools
volumeMounts:
- mountPath: /usr/local/bin/aws
name: custom-tools
subPath: aws
- mountPath: /usr/local/bin/helmsetup.sh
name: plugin-scripts
subPath: helmsetup.sh
readOnly: true
- mountPath: /usr/local/bin/helmtemplate.sh
name: plugin-scripts
subPath: helmtemplate.sh
readOnly: true
server:
config:
configManagementPlugins: |
- name: helm_aws_oci
init:
command: ["/bin/sh", "-c", "/usr/local/bin/helmsetup.sh"]
generate:
command: ["/bin/sh", "-c", "/usr/local/bin/helmtemplate.sh"]
Add scripts configmap under templates/plugin-scripts.yaml:
---
apiVersion: v1
kind: ConfigMap
metadata:
name: plugin-scripts
data:
helmsetup.sh: |
#!/bin/sh
export HELM_EXPERIMENTAL_OCI=1
aws ecr get-login-password --region us-west-2 | helm registry login --password-stdin --username AWS "$(aws sts get-caller-identity --query Account --output text).dkr.ecr.us-west-2.amazonaws.com"
helm dependency build
helmtemplate.sh: |
#!/bin/sh
export HELM_EXPERIMENTAL_OCI=1
helm template --api-versions ${KUBE_API_VERSIONS} -n ${ARGOCD_APP_NAMESPACE} ${ARGOCD_APP_NAME} . --include-crds ${HELM_VALUES_FILES} ${HELM_ARGS}
In application def, replace helm section with plugin:
apiVersion: argoproj.io/v1alpha1
kind: Application
*******
source:
path: ./lacework/helm_base
plugin:
env:
- name: HELM_VALUES_FILES
value: --values ../overlays/a0000000027/globals.yaml --values ../overlays/flags.yaml
--values ../overlays/a0000000027/values.yaml --values ../overlays/a0000000027/hotfix.yaml
- name: HELM_ARGS
value: --set lacework.lacework-agent.laceworkConfig.accessToken='**************************'
--set lacework.lacework-agent.laceworkConfig.env='mushu.falkor.rocks'
--set lacework.lacework-agent.laceworkConfig.kubernetesCluster='a0000000027'
name: helm_aws_oci
@pthornton, sorry for replying late.
I am working with 3.26.12 and yes, I called this myself:
aws ecr get-login-password --region us-east-1 | helm registry login --password-stdin --username AWS "$(aws sts get-caller-identity --query Account --output text).dkr.ecr.us-east-1.amazonaws.com"
I would try to check this week if something changed during the last two weeks in the code. I could also submit a PR to somehow fix this, but I am unsure of how we would like to solve this. I'll think about it and will try to make a proposal, if I can.
Good that you have the plugin working, it looks like a lot of work and I would rather have argo-cd support AWS OCI itself!
@mfbieber Any updates here? We are running into the same issue.
Version:
$ argocd version
argocd: v2.3.3+07ac038.dirty
BuildDate: 2022-03-30T05:20:18Z
GitCommit: 07ac038a8f97a93b401e824550f0505400a8c84e
GitTreeState: dirty
GoVersion: go1.18
Compiler: gc
Platform: darwin/amd64
argocd-server: v2.2.5+8f981cc
BuildDate: 2022-02-05T01:33:25Z
GitCommit: 8f981ccfcf942a9eb00bc466649f8499ba0455f5
GitTreeState: clean
GoVersion: go1.16.11
Compiler: gc
Platform: linux/amd64
Ksonnet Version: v0.13.1
Kustomize Version: v4.2.0 2021-06-30T22:49:26Z
Helm Version: v3.7.1+g1d11fcb
Kubectl Version: v0.22.2
Jsonnet Version: v0.17.0
Repo was added successfully.
$ argocd repo list
TYPE NAME REPO INSECURE OCI LFS CREDS STATUS MESSAGE
helm standard-service xxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com false true false true Successful
We are using this OCI repo as a subchart dependency, also our ArgoCD application is generated using ApplicationSet
name: sample-service
version: "0.1.0"
kubeVersion: ">= 1.19.0-0"
description: chart to deploy a ccp-sample-service
dependencies:
- name: standard-service
repository: oci://xxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com
version: 1.0.0
This is the error I get
rpc error: code = Unknown desc = Manifest generation error (cached): `helm repo add --username ****** --password ****** standard-service oci://xxxxxxxx.dkr.ecr.us-east-1.amazonaws.com` failed exit status 1: Error: looks like "oci://xxxxxxxx.dkr.ecr.us-east-1.amazonaws.com" is not a valid chart repository or cannot be reached: object required
Hi guys, sorry, I don't manage to work on this efficiently at the moment (too much going on).
In general, we abandoned the OCI approach in our projects, because it didn't seem to be properly supported with ECR - at least when we checked. Also what we really disliked is the workaround around logging in every x hours with a workaround CronJob: https://github.com/argoproj/argo-cd/issues/6062#issuecomment-859382021
I believe this issue is now fixed in v2.5.4, by: https://github.com/argoproj/argo-cd/pull/11327
@hedasaurabh / @pthornton can you confirm?
Closing, we've been using ECR Helm charts in prod for quite some time.
If you are trying to resolve an environment-specific issue or have a one-off question about the edge case that does not require a feature then please consider asking a question in argocd slack channel.
Checklist:
argocd version
.Describe the bug Application Chart:
All applications referencing AWS oci charts fail after upgrade. It appears "helm registry login" is only called when add the helm repo using "helm repo add" from cronjob. The "helm dependency update" fails and no login messages appear in repoServer logs.
Error message from UI:
Repo server log:
If I bash into the repo server and do "helm registry login" and then "help dependency build" in the /tmp/https___gitlab.com_somewhere_prt_example-apps_mushu-dev/react-frontend-service/helm_base folder, it successfully pulls the OCI helm chart. So no permission issues. I'm using username/password/url from the argocd cli helm repo add created kubernetes secret repo-4159968007.
This also happens if I upgrade to 2.1.7 instead of 2.2.2. Post in argocd slack also: slack
To Reproduce
Add Application using chart with reference to AWS OCI helm chart.
Expected behavior
Helm charts should pull from AWS OCI after successful "helm registry login" executed before running "helm dependency update".
Screenshots
If applicable, add screenshots to help explain your problem.
Version
Logs