bucc up --cpi docker and Mac docker-desktop certificate problems

[dhoffi]

I spend the last days to get up bucc gainst docker-desktop and finally have to give up if some of you do not have brilliant ideas.

Here's what I did:

• running bucc up --cpi docker first failed because of missing certs then because the mac docker-desktop tcp port is not exposed to the mac host

so I did create certs against *.my.dev domain, configured docker-desktop to use these certs and use a docker create networked network and socat openssl-listen:$SECURE_PORT,fork,reuseaddr,cert=$CERTSDIRHOST/$TLSCERT,cafile=$CERTSDIRHOST/$TLSCACERT,key=$CERTSDIRHOST/$TLSKEY UNIX-CONNECT:/var/run/docker.sock and dnsmasqed my.dev to point at localhost.

But in the end bucc up with docker_host: tcp://something.my.dev:2376 complained on unreachable internal_ip:

Deploying:
  Creating instance 'bosh/0':
    Waiting until instance is ready:
      Post https://mbus:<redacted>@172.19.0.3:6868/agent: dial tcp 172.19.0.3:6868: i/o timeout

Exit code 1

So my gazillionsed try was the following:

• self signed certs versus the faked host.docker.internal which docker-desktop provides to reach the host
• socat from above to make the docker api port available on the mac host via ssl

created a docker bridged network:

docker network inspect buccDockerNetwork | jq -r .[0].IPAM.Config
[
  {
    "Subnet": "172.19.0.0/16",
    "Gateway": "172.19.0.1"
  }
]

• ran a plain ubuntu container mounting the certs and the dockerd socket of the mac host:

NAME=jumphost
USER=hoffi
PASS=hoffi
NETWORK="--network=buccDockerNetwork"
CERTSDIRHOST="/Users/hoffmd9/devTools/docker/certs"
CERTSDIRDOCKER="/certs"
docker run --name=${NAME} ${NETWORK} -t -d \
    -v /Users/hoffmd9/.bosh:/home/.bosh \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v $CERTSDIRHOST:$CERTSDIRDOCKER \
    ubuntu:18.04

• created a user there and installed some packages (including ruby-full)
• installed docker-ce into the ubuntu image by mounting the host dockerd socket (as you can see above)

vars.yml

director_name: buccdocker
# alias: bucc
docker_host: tcp://host.docker.internal:2376
internal_cidr: 172.19.0.0/16
internal_gw: 172.19.0.1
internal_ip: 172.19.0.3
network: buccDockerNetwork
docker_tls:
  <ca cert and key against domain host.docker.internal here...>

on a bucc up --cpi docker I now finally get pretty far, but still not working:

Starting registry... Finished (00:00:00)
Uploading stemcell 'bosh-warden-boshlite-ubuntu-xenial-go_agent/170.9'... Failed (00:00:00)
Stopping registry... Finished (00:00:00)
Cleaning up rendered CPI jobs... Finished (00:00:00)

creating stemcell (bosh-warden-boshlite-ubuntu-xenial-go_agent 170.9):
  CPI 'create_stemcell' method responded with error: CmdError{"type":"Bosh::Clouds::CloudError","message":"Importing stemcell from '/home/hoffi/.bosh/installations/996584b1-4d10-4669-4b6e-be684f041e8f/tmp/stemcell-manager729800701/image': Starting image import: An error occurred trying to connect: Post https://host.docker.internal:2376/v1.22/images/create?fromSrc=-\u0026message=bosh\u0026repo=bosh.io%!F(MISSING)stemcells\u0026tag=img-81bb052b-b4cc-42ed-5767-38bf79f0918f: x509: certificate is not authorized to sign other certificates","ok_to_retry":false}

Exit code 1

or in short:

Post https://host.docker.internal:2376/v1.22/images/create?
fromSrc=-\u0026message=bosh\u0026repo=bosh.io%<redacted>:
x509: certificate is not authorized to sign other certificates"

any ideas? or is this just plain impossible?

[rkoster]

We are currently working on adding support for the docker-desktop cpi (just docker cpi with defaults configured to work with docker-desktop). https://github.com/starkandwayne/bucc/tree/develop/ops/cpis/docker-desktop

[rkoster]

it is ready as is, we only need a new version of concourse with this fix: https://github.com/concourse/concourse-bosh-release/pull/31/files

[ramonskie]

latest concourse is already implemented