Lifting bitbucket pipelines runner limitations

The scripts and services in this repository are intended for lifting the resource limitations imposed on self-hosted Bitbucket Pipelines Linux Docker runners.

Details

As of 2024, Bitbucket Cloud infrastructure runners have a configurable CPU count and memory amount via the `size` option, whereas self-hosted Linux Docker runners are fixed to 4 CPU's:	`size`	Bitbucket Cloud	Self-hosted
`1x`	4 vCPU, 4 GB	4 CPU, 4 GB	∞ CPU, 1 TB *
`2x`	4 vCPU, 8 GB	4 CPU, 8 GB	∞ CPU, 1 TB *
`4x`	8 vCPU, 16 GB	4 CPU, 16 GB	∞ CPU, 1 TB *
`8x`	16 vCPU, 32 GB	4 CPU, 32 GB	∞ CPU, 1 TB *

* Limited by the capabilities of the host.

Cost example

With 1000 Build minutes on Bitbucket Cloud infrastructure being sold for $10 and multiplied with `size`, this can lead to huge costs. Consider the following cost example assuming one runner being busy for one month runtime:	Hetzner Cloud self-hosted	Bitbucket Cloud infrastructure
~$5 (CX22)	$432 (`1x`)
~$10 (CX32)	$864 (`2x`)
~$20 (CX42)	$1728 (`4x`)
~$40 (CX52)	$3456 (`8x`)

Setup

crictl as container runtime (e. g. using hetzner-k3s and runners-autoscaler)

hetzner-k3s_cluster_config.yaml

...
  post_create_commands:
  - git clone https://github.com/tramseyer/bitbucket-pipelines-runner-unlimited.git
  - cd bitbucket-pipelines-runner-unlimited
  - ./setup.sh crictl
...

docker as container runtime (e. g. using a dedicated Linux Docker runner)

git@github.com:tramseyer/bitbucket-pipelines-runner-unlimited.git
cd bitbucket-pipelines-runner-unlimited
./setup.sh docker

kubernetes (e. g. using Docker-based runner on Kubernetes or Autoscaler for Runners on Kubernetes)

kustomize/base/cm-job-template.yaml / config/runners-autoscaler-cm-job.template.yaml

...
                - name: runner
                  ...
                - name: docker
                  ...
                - name: docker-cpu-quota
                  image: docker:cli
                  command: ["/bin/sh", "-c", "wget -qO- https://raw.githubusercontent.com/tramseyer/bitbucket-pipelines-runner-unlimited/master/docker-cpu-quota.sh | sh"]
                  volumeMounts:
                    - name: var-run
                      mountPath: /var/run
                - name: docker-memory-limit
                  image: docker:cli
                  command: ["/bin/sh", "-c", "wget -qO- https://raw.githubusercontent.com/tramseyer/bitbucket-pipelines-runner-unlimited/master/docker-memory-limit.sh | sh"]
                  volumeMounts:
                    - name: var-run
                      mountPath: /var/run
...

Monitoring of services

on kubernetes node

systemctl status --no-pager crictl-{cpu-quota,memory-limit}

on dedicated Linux Docker runner

systemctl status --no-pager docker-{cpu-quota,memory-limit}

tramseyer / bitbucket-pipelines-runner-unlimited

readme