Add kubernetes manifest file with the script to remove bb-runner limits

Hey buddy,

Thanks a lot to for sharing the hack to remove the crazy resource limits exposed on the bb-runners. This is working great on our infrastructure.

I though to share our modified kubernetes manifest file for anyone else who wants to use this on their k8s cluster without modifying the underlying AMI.

Here is the Kubernetes manifest file which includes the script to remove the bb-runner limits:

This is based on the official bb k8s setup -> https://support.atlassian.com/bitbucket-cloud/docs/deploying-the-docker-based-runner-on-kubernetes/

apiVersion: v1
kind: List
items:
  - apiVersion: v1
    kind: ConfigMap
    metadata:
      name: resource-limit-remover-script
      namespace: bitbucket-runner
    data:
      resource-limit-remover.sh: |
        #!/bin/sh

        # The scripts in this configmap is intended for lifting the resource limitations
        # imposed on self-hosted Bitbucket Pipelines Linux Docker runners.
        # https://jira.atlassian.com/browse/BCLOUD-21403
        # https://github.com/tramseyer/bitbucket-pipelines-runner-unlimited

        MEMORY=$((1024 * 1024 * 1024 * 1024)) # 1TB
        while true
        do
          if [ -z "$(docker ps -q)" ]; then
            echo "No Docker containers/builds are running. Exiting gracefully."
          else
            MEMORY_OUTPUT=$(docker inspect --format="{{.Id}} {{.Name}} {{.HostConfig.Memory}} {{.HostConfig.MemorySwap}}" $(docker ps -aq) | grep ".*_build" | grep -v "_build\s$MEMORY")
            if [ $? -eq 0 ]; then
                echo "Found *_build container with different memory limit: $MEMORY_OUTPUT"
                ID="${MEMORY_OUTPUT%% *}"
                docker update --memory="$MEMORY" --memory-swap="$MEMORY" "$ID"
                docker inspect --format "{{.Id}} {{.HostConfig.Memory}} {{.HostConfig.MemorySwap}}" "$ID"
            fi
            CPU_OUTPUT=$(docker inspect --format="{{.Id}} {{.Name}} {{.HostConfig.CpuQuota}}" $(docker ps -aq) | grep ".*_build\s400000")
            if [ $? -eq 0 ]; then
                echo "Found *_build container with active CPU quota: $CPU_OUTPUT"
                ID="${CPU_OUTPUT%% *}"
                docker update --cpu-quota=-1 "$ID"
                docker inspect --format "{{.Id}} {{.HostConfig.CpuQuota}}" "$ID"
            fi
            echo "Sleeping for 10 sec"
          fi
          sleep 10
        done
  - apiVersion: v1
    kind: Secret
    metadata:
      name: runner-oauth-credentials
#      labels:
#        accountUuid: # Add your account uuid without curly braces to optionally allow finding the secret for an account
#        repositoryUuid: # Add your repository uuid without curly braces to optionally allow finding the secret for a repository
#        runnerUuid: # Add your runner uuid without curly braces to optionally allow finding the secret for a particular runner
    data:
      oauthClientId: # add your base64 encoded oauth client id here
      oauthClientSecret: # add your base64 encoded oauth client secret here
  - apiVersion: batch/v1
    kind: Job
    metadata:
      name: runner
    spec:
      template:
#        metadata:
#          labels:
#            accountUuid: # Add your account uuid without curly braces to optionally allow finding the pods for an account
#            repositoryUuid: # Add your repository uuid without curly braces to optionally allow finding the pods for a repository
#            runnerUuid: # Add your runner uuid without curly braces to optionally allow finding the pods for a particular runner
        spec:
          containers:
            - name: runner
              image: docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner
              env:
                - name: ACCOUNT_UUID
                  value: # Add your account uuid here
                - name: REPOSITORY_UUID
                  value: # Add your repository uuid here
                - name: RUNNER_UUID
                  value: # Add your runner uuid here
                - name: OAUTH_CLIENT_ID
                  valueFrom:
                    secretKeyRef:
                      name: runner-oauth-credentials
                      key: oauthClientId
                - name: OAUTH_CLIENT_SECRET
                  valueFrom:
                    secretKeyRef:
                      name: runner-oauth-credentials
                      key: oauthClientSecret
                - name: WORKING_DIRECTORY
                  value: "/tmp"
              volumeMounts:
                - name: tmp
                  mountPath: /tmp
                - name: docker-containers
                  mountPath: /var/lib/docker/containers
                  readOnly: true # the runner only needs to read these files never write to them
                - name: var-run
                  mountPath: /var/run
            - name: docker-in-docker
              image: docker:20.10.5-dind
              securityContext:
                privileged: true # required to allow docker in docker to run and assumes the namespace your applying this to has a pod security policy that allows privilege escalation
              volumeMounts:
                - name: tmp
                  mountPath: /tmp
                - name: docker-containers
                  mountPath: /var/lib/docker/containers
                - name: var-run
                  mountPath: /var/run
                - name: script-volume
                  mountPath: /scripts
            - name: docker-resource-limit-remover
              image: docker:cli
              command: ["/bin/sh", "-c", "/scripts/resource-limit-remover.sh"]
              volumeMounts:
                - name: script-volume
                  mountPath: /scripts
                - name: tmp
                  mountPath: /tmp
                - name: docker-containers
                  mountPath: /var/lib/docker/containers
                  readOnly: true
                - name: var-run
                  mountPath: /var/run

          restartPolicy: OnFailure # this allows the runner to restart locally if it was to crash
          volumes:
            - name: tmp # required to share a working directory between docker in docker and the runner
            - name: docker-containers # required to share the containers directory between docker in docker and the runner
            - name: var-run # required to share the docker socket between docker in docker and the runner
            - name: script-volume
              configMap:
                name: resource-limit-remover-script
                defaultMode: 0777
        # backoffLimit: 6 # this is the default and means it will retry upto 6 times if it crashes before it considers itself a failure with an exponential backoff between
        # completions: 1 # this is the default the job should ideally never complete as the runner never shuts down successfully
        # parallelism: 1 # this is the default their should only be one instance of this particular runner

tramseyer / bitbucket-pipelines-runner-unlimited

Add kubernetes manifest file with the script to remove bb-runner limits #1