Breaking changes in latest package bundles for credential-provider-package

[czomo]

What happened: Any version of package bundle above v1-27-128 are not usable because of multiple issues. Should we keep those faulty packages in registry? Is there any end2end test that could detect that in future?

v1-27-137   other   7 days ago   public.ecr.aws/eks-anywhere/...es-bundles:v1-27-137    60.8 KB > doesn't work, image of anywhere-package controller works fine, see log_1
v1-27-134   other   14 days ago  public.ecr.aws/eks-anywhere/...es-bundles:v1-27-134    60.7 KB > faulty secret, propably because of helm chart
v1-27-130   other   17 days ago  public.ecr.aws/eks-anywhere/...es-bundles:v1-27-130    60.7 KB > wrong helm app version which causing imagepullbackoff for package controler and refresher, see log_2
v1-27-129   other   17 days ago  public.ecr.aws/eks-anywhere/...es-bundles:v1-27-129    60.7 KB > wrong helm app version which causing imagepullbackoff for package controler and refresher, see log_2
v1-27-128   other   2 months ago     public.ecr.aws/eks-anywhere/...es-bundles:v1-27-128 > works fine

log_1

2023-11-02T10:52:24.780Z    ECRCredInjector    Failed to inject ECR credential to docker config    {"error": "operation error ECR: GetAuthorizationToken, failed to sign request: failed to retrieve credentials: failed to refresh cached credentials, static credentials are empty"}
github.com/aws/eks-anywhere-packages/pkg/registry.(*ECRCredInjector).Run
    github.com/aws/eks-anywhere-packages/pkg/registry/ecr_cred_injector.go:56

log_2

eksa-packages  eks-anywhere-packages             8        failed    eks-anywhere-packages-0.0.0-8862036270224f2a6b8d6ecd455b6b1fa1084619              v0.0.0-8862036270224f2a6b8d6ecd455b6b1fa1084619    

What you expected to happen: eks-anywhere-packages shouldn't be published with such breaking changes How to reproduce it (as minimally and precisely as possible):

1. Using 0.17.4 eks-anywhere install k8s 1.27 using tinkerbell provider
2. Create eks-anywhere role along with anchor, follow https://anywhere.eks.amazonaws.com/docs/packages/credential-provider-package/iam_roles_anywhere/#prerequisites
3. Create aws-config secret in eks-packages ns

   [default]
   region = eu-west-1
   credential_process = aws_signing_helper credential-process --certificate /var/lib/kubelet/pki/kubelet-client-current.pem --private-key /var/lib/kubelet/pki/kubelet-client-current.pem --profile-arn $PROFILE_ARN --role-arn $ROLE_ARN --trust-anchor-arn $TRUST_ANCHOR_ARN

4. Add package to download from private ECR registry

   apiVersion: packages.eks.amazonaws.com/v1alpha1
   kind: Package
   metadata:
   name: my-credential-provider-package
   namespace: eksa-packages-eksa
   annotations:
   "helm.sh/resource-policy": keep
   "anywhere.eks.aws.com/internal": "true"
   spec:
   packageName: credential-provider-package
   targetNamespace: eksa-packages
   config: |-
   tolerations:
     - key: "node-role.kubernetes.io/master"
       operator: "Exists"
       effect: "NoSchedule"
     - key: "node-role.kubernetes.io/control-plane"
       operator: "Exists"
       effect: "NoSchedule"
   sourceRegistry: public.ecr.aws/eks-anywhere
   credential:
     - matchImages:
       - 000000000000.dkr.ecr.eu-west-2.amazonaws.com
       profile: "default"
       secretName: aws-config
       defaultCacheDuration: "12h"

5. Verify you have latest version of packagebundle in packagebundlecontroller
6. Create pod with image from 000000000000.dkr.ecr.eu-west-2.amazonaws.com registry
7. ImagePullBackOff should be logged from kubelet

Anything else we need to know?: We also checked latest v1-28 with k8s 1.28 and it also experience issues the same as v1-27-137

Environment: k8s 1.27, tinker provisioner with bare bone nodes, ubuntu 22.04 ami

• EKS Anywhere Release: 0.17.4
• EKS Distro Release: -

[chrisdoherty4]

Thanks for the report @czomo. We're aware of the problem and will fix asap.

[czomo]

@chrisdoherty4 Any update from your side? It's not working on v1-27-142 neither of any 1.28-*.

[joeto0]

Is that fixed? We are hitting same behavior with 1.28.7.

[mitalipaygude]

Can you confirm the OS you are using @joeto0 ? Is it Bottlerocket?

Also, @czomo can you confirm the OS you are using as well? Its Ubuntu right?

[joeto0]

yes, bottlerocket.

On Wed, Apr 24, 2024 at 6:38 PM Mitali Paygude @.***> wrote:

Can you confirm the OS you are using @joeto0 https://github.com/joeto0 ? Is it Bottlerocket?

Also, @czomo https://github.com/czomo can you confirm the OS you are using as well? Its Ubuntu right?

— Reply to this email directly, view it on GitHub https://github.com/aws/eks-anywhere-packages/issues/1024#issuecomment-2075963870, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC4HXO5BOEOOISNGQTGLBETY7AX6PAVCNFSM6AAAAAA64K3QD2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZVHE3DGOBXGA . You are receiving this because you were mentioned.Message ID: @.***>

[czomo]

Also, @czomo can you confirm the OS you are using as well? Its Ubuntu right?

yes, Ubuntu we find out that in our case faulty was the host Path, as for now we stopped using packages in favour of predefined daemonset '''

• hostPath: path: /etc/sysconfig/kubelet type: FileOrCreate '''