Running Airbyte Open source in Docker Compose

[Chammar37]

Platform Version

0.61.0

What step the error happened?

During the Sync

Relevant information

Hello everyone,

I am looking to deploy Airbyte's open source solution in a docker container to easily deploy on a few other computers. Slowly I have understood how Airbyte functions and I am coming across some low level kernel issues (cgroups and docker daemon issues). More specifically:

When airbyte is trying to create docker containers for the connection between source and destination I am getting the following error :

cannot enter cgroupv2 "/sys/fs/cgroup/docker" with domain controllers -- it is in an invalid state: unknown.

This is the result of the following command :

docker run --rm --init -i -w /data/1/0 --log-driver none --name source-google-sheets-read-1-0-jnany -e CONCURRENT_SOURCE_STREAM_READ=false --network host -v airbyte_workspace:/data -v oss_local_root:/local -e DEPLOYMENT_MODE=OSS -e WORKER_CONNECTOR_IMAGE=airbyte/source-google-sheets:0.5.1 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e FIELD_SELECTION_WORKSPACES= -e USE_STREAM_CAPABLE_STATE=true -e AIRBYTE_ROLE=dev -e WORKER_ENVIRONMENT=DOCKER -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e OTEL_COLLECTOR_ENDPOINT=http://host.docker.internal:4317 -e FEATURE_FLAG_CLIENT=config -e AIRBYTE_VERSION=0.61.0 -e WORKER_JOB_ID=1 --cpus=1 --memory-reservation=1Gi --memory=2Gi airbyte/source-google-sheets:0.5.1 read --config source_config.json --catalog source_catalog.json

Note: I tried running the command on the container without "--cpus=1 --memory-reservation=1Gi --memory=2Gi" and it worked without an issue .

Has anyone else had to solve this problem here? I understand it might be a Docker-in-Docker problem but looking for help anywhere at this point.

Relevant log output

docker run --rm --init -i -w /data/1/0 --log-driver none --name source-google-sheets-read-1-0-jnany -e CONCURRENT_SOURCE_STREAM_READ=false --network host -v airbyte_workspace:/data -v oss_local_root:/local -e DEPLOYMENT_MODE=OSS -e WORKER_CONNECTOR_IMAGE=airbyte/source-google-sheets:0.5.1 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e FIELD_SELECTION_WORKSPACES= -e USE_STREAM_CAPABLE_STATE=true -e AIRBYTE_ROLE=dev -e WORKER_ENVIRONMENT=DOCKER -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e OTEL_COLLECTOR_ENDPOINT=http://host.docker.internal:4317 -e FEATURE_FLAG_CLIENT=config -e AIRBYTE_VERSION=0.61.0 -e WORKER_JOB_ID=1 --cpus=1 --memory-reservation=1Gi --memory=2Gi airbyte/source-google-sheets:0.5.1 read --config source_config.json --catalog source_catalog.json

Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: cannot enter cgroupv2 "/sys/fs/cgroup/docker" with domain controllers -- it is in an invalid state: unknown.

[marcosmarxm]

What OS are you using?

[Chammar37]

My local is MacOS Sonoma

On the Dockerfile im pulling FROM this image python:3.12-bookworm (Debian based) :

FROM python:3.12-bookworm as airbyte

[Chammar37]

I also want to note, running the ./run-ab-platform.sh script on my local works without issue (running MacOS as stated above). Something about adding another layer of container to the environments breaks the required cgroups permissions. I read here that :

if devices is standing on a line on its own, you’re good. If you see that another control group is on the same line, Docker-in-Docker won’t work

Maybe this is what is causing the issue? Not exactly sure how to test this though other that cat /proc/pid_here/cgroup