search

koslib / helm-eks-action

The simplest Github Action for executing Helm commands on EKS - cluster authentication included
MIT License
59 stars 61 forks source link

kubectl commands don't work #41

Open riontric opened 1 year ago

riontric commented 1 year ago

Hi, pipelines are dropped with error: /usr/bin/kubectl: line 1: syntax error: unexpected redirection if i use kubectl in command line. All worked on friday.

koslib commented 1 year ago

Hi, issued release https://github.com/koslib/helm-eks-action/releases/tag/v1.25.3 over the weekend bumping versions of kubectl, awscli and helm. Also tested this out in plenty of my own pipelines and it works okay. Could you post some more details about the structure, jobs and steps of your workflow to take a look?

riontric commented 1 year ago 
      - name: Helm deploy
        uses: koslib/helm-eks-action@master
        env:
          KUBE_CONFIG_DATA: ${{ secrets.KUBE_CONFIG_DATA_DEV }}
        with:
          plugins: |
            https://github.com/jkroepke/helm-secrets
          command: |
            aws ecr get-login-password --region us-east-1 | helm registry login --username AWS --password-stdin ${{ env.HELM_PRIVATE_REGISTRY }} && \
              kubectl get pods && helm ls && \
              helm upgrade chart-name -i --atomic -f .env.yaml oci://${{ env.HELM_PRIVATE_REGISTRY }}/helm-chart --version 0.1.9 \
              helm ls && kubectl get pods

this release -> https://github.com/koslib/helm-eks-action/releases/tag/v1.25.2 work for me.

romanlobai commented 1 year ago

Hey, guys! I have the same issue. On Friday lib worked, now - doesn't work.

error: /usr/bin/kubectl: line 1: syntax error: unexpected redirection

koslib commented 1 year ago

Hmm it might have to do with the fact I started building the base docker image with buildx https://github.com/koslib/helm-eks-action/blob/master/.github/workflows/dockerhub.yaml#L23-L28. Some googling I did indicated this might be a multi-arch issue.

I am contemplating fixing it to linux/amd64 platforms in the build step. Any thoughts or other ideas?

koslib commented 1 year ago

I just run this and built the base docker image with buildx for linux/amd64 platforms as I suspect the multiarch build might have to do something with this.

Could you let me know how it feels now?

Sharlie89 commented 1 year ago

Hi, I started to have some errors related to auth API, since you upgrade the docker image (koslib/ga-helm-eks) version from 2.0 to 2.1 & 2.2

If I use koslib/ga-helm-eks:2.0 everything works fine, but if I increase it fails. Have anyone of you experimented this?

The error is related to auth API v1alpha1.

ruankotovich commented 1 year ago

Hello there

The same happening to me today:

      uses: koslib/helm-eks-action@master
      with:
          command: |
            kubectl create secret generic $NAMESPACE-$APP_NAME -n $NAMESPACE \
            --save-config --dry-run=client \
            --from-env-file .env -o yaml | kubectl apply -f -
/usr/bin/kubectl: line 1: syntax error: unexpected redirection

@koslib

koslib commented 1 year ago

@riontric @romanlobai @Sharlie89 I have two potential ideas for this issue:

  1. It might be caused of the build arch of the image
  2. It might be related to the kubectl version upgrade (although I really don't see how - yet)

Tried updating the koslib/ga-helm-eks:2.2 docker image to in an attempt to address idea no 1. Could you please retry running your actions and see if it has helped or is it still failing with the same command?

To address idea no 2 please mention your cluster versions and we can try a couple of things then.

Thanks in advance for your help 🙏🏼

koslib commented 1 year ago

@koslib guys this is HUGE. please fix this bug ASAP

@SaharAvr We are not a company, I am a person :)

Since I also saw (and kept a copy of) your previous very very rude comment which you deleted, I am prioritizing support for all the kindly-communicating colleagues who reported their issues and actively participate with input in resolving this issue.

And again, this is an open-source repository where you can freely jump in and help and submit a fixing PR. Thanks for your upcoming contribution to this :)

koslib commented 1 year ago

@koslib It was removed because for some reason I mistaken this to be an AWS repo, in which case I wouldn't have removed my original comment

Now that I realise this is a one man's show, I'm going to remove any usages of this action in my workplace, as this is highly dangerous

Feel free 👋🏼

Sharlie89 commented 1 year ago

Hey @koslib,

I'm gonna test it today with the image 2.2. I've keep this working with image 2.0 but with the new versions of the kubectl, awscli and helm. I got a warning about the AWS api v1alpha1 (when it's supposed to be v1beta1), but hope this give you a clue about what is happening (I also thought that is related to the build arch).

@SaharAvr What's is highly dangerous is to have someone on charge of something who's implementing things that he's no testing and understanding at all. Also, goes to public github repo, demanding disciplinary treatment is, from my point of view, a clear behaviour of a bad person. EVERYONE OF US MAKES MISTAKES, everyone. You can't go to a public site demanding punishment for one error. Also this is a free use action, so I don't know why you feel in the place for demanding this kind of stuff.

Sharlie89 commented 1 year ago

@koslib

Just tested it, doesn't working at all. I'm getting again this error:

Error: Kubernetes cluster unreachable: exec plugin: invalid apiVersion "client.authentication.k8s.io/v1alpha1"

I doesn't know why, but this is happening in one of my GH actions, but it's working on another. Both of them had the same cluster as an endpoint.

koslib commented 1 year ago

@Sharlie89 well, I think I understand what's up - let me explain and please let me know if this makes any sense to you too:

I'll get back to you as I have more updates - hopefully before the weekend! Your time and help is really appreciated 🙏🏼

koslib commented 1 year ago

@Sharlie89 I was actually wrong. Turns out the issue has to do with the awscli@1.24.x version, as k8s upgraded the auth api and the token given from that aws cli version did not support that correctly.

I made a new branch named experimental where:

and tried this out with no issues on my end.

Could you give it a shot as well to confirm it's also ok for your systems before I PR to master? 

- uses: koslib/helm-eks-action@experimental
...

Potentially @riontric and @romanlobai can try this out too and see how that goes?

Thank you all in advance!

Sharlie89 commented 1 year ago

@koslib just tested it, same error. Error: Kubernetes cluster unreachable: exec plugin: invalid apiVersion "client.authentication.k8s.io/v1alpha1" At this point I don't know what is happening. Also I don't know why I'm getting and invalid apiVersion with v1alpha1, when Kubeconfig is pointing to v1beta1. But more weird is the fact that it's works on one job and failing in another.

Sharlie89 commented 1 year ago

Looking through all of this, the only pain point that I see is that all of this started when you changed the way of built the docker image. But this is what I see from my site, can't be sure 100%. I forked your repo when you where setting the docker image koslib/ga-helm-eks:2.0, and this works for me. But not any of the other images above 2.0.

I updated one of my clusters today to 1.22 versions, but no luck also. I ran out of ideas about what can be happening.

koslib commented 1 year ago

I'm trying to piggy back on your train of thought @Sharlie89 .

What this PR did:

In order to eliminate any multi-arch issues later on, I built and pushed locally with --platform linux/amd64, but the issue persisted. Which leaves us with the dependencies updates.

just tested it, same error. Error: Kubernetes cluster unreachable: exec plugin: invalid apiVersion "client.authentication.k8s.io/v1alpha1" At this point I don't know what is happening. Also I don't know why I'm getting and invalid apiVersion with v1alpha1, when Kubeconfig is pointing to v1beta1

In the experimental branch you got this error, which eliminates eg. an outdated aws-iam-authenticator to be the root cause. Therefore maybe a difference between the awscli of the action and your version for generating the kube context data? Can you post more details regarding how you generate your kubeconfig file?

But more weird is the fact that it's works on one job and failing in another.

Could you post how these jobs look like? There must be a difference somewhere.

Finally, you have an EKS at v1.22, right? I'll build a new test cluster mimicking exactly your setup to test this out. Updates to come.

Sharlie89 commented 1 year ago

In this one, is working:


name: Deploy to Sandbox

env:
  AWS_REGION: eu-central-1

on:
  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout                                  
      uses: actions/checkout@v2                       
      with:                                           
        ref: ${{ github.event.pull_request.head.sha }}

    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: ${{ secrets.SBX_AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.SBX_AWS_SECRET_ACCESS_KEY }}
        aws-region: ${{ env.AWS_REGION }}

    - name: Login to Amazon ECR
      id: login-ecr
      uses: aws-actions/amazon-ecr-login@v1

    - name: Build, tag, and push image to Amazon ECR
      env:
        ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
        ECR_REPOSITORY: api-gateway-sandbox
        IMAGE_TAG: ${{ github.sha }}
      run: |
        docker build --build-arg BASE_IMAGE=node:17.6.0-alpine -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .
        docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
    - name: Deploy to Cluster
      uses: koslib/helm-eks-action@master
      env:
        KUBE_CONFIG_DATA: ${{ secrets.SBX_KUBECONFIG }}
      with:
        command: helm upgrade api-gateway --install ./helm/api-gateway --wait --set image.tag=${{ github.sha }} -f ./helm/values/sandbox/values.yaml -n XXXXX

  slack_notification:
    needs: [build]
    if: failure()
    uses: XXXXXX/infra-api-gateway/.github/workflows/slack_notification.yml@develop
    secrets:
      SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}

In this, fails:

  name: Deploy to Sandbox

env:
  AWS_REGION: eu-central-1

on:
  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout                                  
      uses: actions/checkout@v2                       
      with:                                           
        ref: ${{ github.event.pull_request.head.sha }}

    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: ${{ secrets.SBX_AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.SBX_AWS_SECRET_ACCESS_KEY }}
        aws-region: ${{ env.AWS_REGION }}

    - name: Login to Amazon ECR
      id: login-ecr
      uses: aws-actions/amazon-ecr-login@v1

    - name: Build, tag, and push image to Amazon ECR
      env:
        ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
        ECR_REPOSITORY: monolith-sandbox
        IMAGE_TAG: ${{ github.sha }}
      run: |
        docker build --build-arg BASE_IMAGE=node:17.6.0-alpine -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .
        docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
    - name: Build development image, tag, and push image to Amazon ECR
      env:
        ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
        ECR_REPOSITORY: monolith-sandbox
        IMAGE_TAG: ${{ github.sha }}
      run: |
        docker build -f Dockerfile.migrations --build-arg BASE_IMAGE=node:17.6.0-alpine -t ${ECR_REGISTRY}/${ECR_REPOSITORY}:${IMAGE_TAG}-development .
        docker push ${ECR_REGISTRY}/${ECR_REPOSITORY}:${IMAGE_TAG}-development
    - name: Deploy to Cluster
      uses: koslib/helm-eks-action@experimental
      env:
        KUBE_CONFIG_DATA: ${{ secrets.SBX_KUBECONFIG }}
      with:
        command: helm upgrade monolith --install ./helm/monolith --wait --timeout 10m --set image.tag=${{ github.sha }} -f ./helm/values/sandbox/values.yaml -n XXXXX

  slack_notification:
    needs: [build]
    if: failure()
    uses: XXXX/infra-monolith/.github/workflows/slack_notification.yml@develop
    secrets:
      SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
koslib commented 1 year ago

So the master branch works for you but not the experimental one. Does this also mean tag v1.25.4 works? 🤔 Now I'm getting more confused :D But I'll try to find some time during the coming days to try stuff out on demo clusters.

Sharlie89 commented 1 year ago

So the master branch works for you but not the experimental one. Does this also mean tag v1.25.4 works? 🤔 Now I'm getting more confused :D But I'll try to find some time during the coming days to try stuff out on demo clusters.

No no, in the failing job doesn't work with: experimental, master or v1.25.4. Only works with the docker image koslib/ga-helm-eks:2.0.

I've tested this with my forked repo.