(kubectl run failing on pipeline) Connection Refused - Couldn't get current server API group list

[ougabriel]

Error Description Run kubectl apply -f k8s-manifest/deployment.yml --validate=false E0530 08:41:13.702608 1817 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp [::1]:8080: connect: connection refused error: unable to recognize "k8s-manifest/deployment.yml": Get "http://localhost:8080/api?timeout=32s": dial tcp [::1]:8080: connect: connection refused Error: Process completed with exit code 1.

Environment Kubernetes is running on EC2 instance, All codes, kubernetes manifest yaml files and pipeline scripts are pushed to repo

Expected behavior GitActions is expected to connect to kube-apiserver and run the kubectl deployment

Screenshots

Possible Solution Setting up OpenID connect to authenticate GitActions with AWS EC2

---

Using OpenID Connect (OIDC) to authenticate GitHub Actions with AWS for your Kubernetes operations is a more secure and modern approach compared to using static credentials. This method leverages the OIDC token that GitHub Actions provides to authenticate with AWS and obtain temporary security credentials. Here’s a step-by-step guide on how to set this up:

Step 1: Configure OIDC Provider in AWS

1. Create an IAM OIDC Identity Provider:

   • Go to the IAM console in AWS.
   • Under Identity providers, click Add provider.
   • Choose OpenID Connect as the provider type.
   • For the provider URL, enter: https://token.actions.githubusercontent.com.
   • For the audience, you can use sts.amazonaws.com.

2. Create an IAM Role for GitHub Actions:

   • Go to the IAM console.
   • Click on Roles, then Create role.
   • Choose Web identity as the trusted entity type.
   • Select the OIDC provider you just created.
   • For the audience, use sts.amazonaws.com.
   • Click Next: Permissions.
   • Attach the necessary policies for Kubernetes and AWS resources (e.g., Amazon EKS cluster permissions).
   • Click Next: Tags and then Next: Review.
   • Name your role and create it.

Step 2: Add GitHub Repository to IAM Role Trust Policy

1. Edit the Trust Policy of the IAM Role:

   • Go to the IAM console.
   • Click on Roles, then select the role you just created.
   • Under the Trust relationships tab, click Edit trust policy.
   • Add the following JSON to allow your GitHub repository to assume the role:

   {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
          "Federated": "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/token.actions.githubusercontent.com"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
          "StringEquals": {
            "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
            "token.actions.githubusercontent.com:sub": "repo:<GITHUB_USERNAME>/<REPOSITORY_NAME>:ref:refs/heads/<BRANCH_NAME>"
          }
        }
      }
    ]
   }

   Replace <AWS_ACCOUNT_ID>, <GITHUB_USERNAME>, <REPOSITORY_NAME>, and <BRANCH_NAME> with your specific values.

Step 3: Configure GitHub Actions Workflow

1. Create a GitHub Actions Workflow:

   Create a workflow file (e.g., .github/workflows/deploy.yml) in your repository with the following content:

   name: Deploy to Kubernetes
   
   on:
    push:
      branches:
        - main
   
   jobs:
    deploy:
      runs-on: ubuntu-latest
   
      permissions:
        id-token: write
        contents: read
   
      steps:
        - name: Checkout code
          uses: actions/checkout@v2
   
        - name: Configure AWS credentials using OIDC
          uses: aws-actions/configure-aws-credentials@v2
          with:
            role-to-assume: arn:aws:iam::<AWS_ACCOUNT_ID>:role/<ROLE_NAME>
            aws-region: us-west-2
   
        - name: Set up kubectl
          run: |
            curl -LO "https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl"
            chmod +x ./kubectl
            sudo mv ./kubectl /usr/local/bin/kubectl
   
        - name: Configure kubeconfig
          run: |
            mkdir -p ~/.kube
            echo "${{ secrets.KUBECONFIG }}" > ~/.kube/config
   
        - name: Deploy to Kubernetes
          run: kubectl apply -f k8s/deployment.yaml

Explanation

• Permissions: The id-token: write permission allows the workflow to request an OIDC token.
• Configure AWS Credentials using OIDC: This step configures AWS credentials dynamically using the OIDC token issued by GitHub Actions.
• Set up kubectl: This installs kubectl on the GitHub runner.
• Configure kubeconfig: This configures kubectl to use your Kubernetes cluster.
• Deploy to Kubernetes: This runs the kubectl apply command to deploy your Kubernetes manifests.

Additional Steps

• Ensure your kubeconfig file in GitHub Secrets is correctly formatted and has the correct permissions.
• Test the GitHub Actions workflow to ensure it can successfully authenticate and interact with your Kubernetes cluster using the OIDC provider.

By setting up GitHub Actions with OIDC for AWS authentication, you improve the security and manageability of your CI/CD pipeline, avoiding the need to manage long-lived AWS credentials.