search

abiosoft / colima

Container runtimes on macOS (and Linux) with minimal setup
MIT License
19.02k stars 383 forks source link

How to add trust unknown ssl authority or manually download and use iso #131

Closed matejsp closed 2 years ago

matejsp commented 2 years ago

Is it possible to skip ssl check? We are using antivirus with MITM ssl certificate.

colima start --cpu 6 --memory 8 --disk 60 | more
INFO[0000] starting colima                              
INFO[0000] starting ...                                  context=vm
time="2022-01-18T13:17:29+01:00" level=info msg="Using the existing instance \"colima\""
time="2022-01-18T13:17:29+01:00" level=info msg="Attempting to download the image from \"https://github.com/abiosoft/alpine-lima/releases/download/colima-v0.3.2/alpine-lima-clm-3.14.3-x86_64.iso\"" digest="sha512:875482176ff2f43bf9472f84137d8b9a56d692ae19243436079c7f2f78cb6b13576601eef9102aaea796629215fc60308e69b590c81fe4139e9e84f1ec4a57f5"
time="2022-01-18T13:17:29+01:00" level=fatal msg="failed to download the image, attempted 2 candidates, errors=[unsupported arch: \"aarch64\" failed to download \"https://github.com/abiosoft/alpine-lima/releases/download/colima-v0.3.2/alpine-lima-clm-3.14.3-x86_64.iso\": Get \"https://objects.githubusercontent.com/github-production-release-asset-2e65be/422815134/912100a2-63cc-4bb2-ac99-e0940a5af5f5?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220118%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220118T121636Z&X-Amz-Expires=300&X-Amz-Signature=3d7b0a81026ed973ff0a64b8d9141f6104dcc775c35d2300dffc4c1283379e6d&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=422815134&response-content-disposition=attachment%3B%20filename%3Dalpine-lima-clm-3.14.3-x86_64.iso&response-content-type=application%2Foctet-stream\": x509: certificate signed by unknown authority]"
FATA[0000] error starting vm: error at 'starting': exit status 1
V3ckt0r commented 2 years ago

it doesn't seem that colima is respecting host certs, such as /etc/docker/certs.d/*. Using Colima with a private registry gives me this

docker login <private_reg>
...
Error response from daemon: Get "<private_reg>": x509: certificate signed by unknown authority

This is with latest (as of today)

colima version
colima version HEAD-272db47
git commit: 272db4732b90390232ed9bdba955877f46a50552

runtime: docker
arch: x86_64
client: v20.10.12
server: v20.10.11
matejsp commented 2 years ago

I solved this by adding ssl inspection cert inside Keychain Access -> Certificates. I also needed to select Always Trust.

kkasberg commented 2 years ago

I have the same issue, but it is because my corporate traffic is proxied and has tls inspection. The Corp cert is installed in the kay chain -> certs, but it still won’t pull docker images from dockerhub

kkasberg commented 2 years ago

-- bump

How can I add my corp cert into the VM so that it can pull down images?

huckeduster commented 2 years ago

Would also be interested in a solution here. Trying to add CA certificate to Colima VM and restarting docker daemon (just to have a proof of concept) did not work for me. @matejsp Could you elaborate a bit on the way you solved that as mentioned in https://github.com/abiosoft/colima/issues/131#issuecomment-1015398218?

jandubois commented 2 years ago

I would copy the certificate to /usr/local/share/ca-certificates and then run /usr/sbin/update-ca-certificates, followed by rc-service docker restart (based on my knowledge of the Alpine image; not sure if colima needs something in addition).

abiosoft commented 2 years ago

Would also be interested in a solution here. Trying to add CA certificate to Colima VM and restarting docker daemon (just to have a proof of concept) did not work for me. @matejsp Could you elaborate a bit on the way you solved that as mentioned in #131 (comment)?

@huckeduster where are the certificates stored on the macOS host?

matejsp commented 2 years ago

@abiosoft To get this working I tried various things. But the one that worked on Mac was to open Keychain Access navigate to login on the left ... and on the right select Certificates import ssl introspection cert that you use double click on ssl introspection cert expand Trust select Always Trust for all certificates and confirm with your credentials

After that I was able to download iso using colima (that is using curl behind the scenes).

huckeduster commented 2 years ago

@abiosoft The certs are probably stored in keychain, but I can also export them (SSL cert of registry, CA and Root CA certs), convert them to whatever necessary (pem, crt, cer) via openssl but I did not manage to add them to colima. And probably that would anyway only work until next colima stop/start (adding unpreferred "insecure-registries" : ["xxx:5000"] to docker daemon config did also not persist a colima restart). So I would still be interested in @matejsp's solution. What introspection cert are you talking about?

matejsp commented 2 years ago

Antivirus in our company monitors by intercepting SSL and HTTPS traffic using self generated certificate. This intermediate certificate is used for issuing temporary certificates that have the same data as original (but different key). It is disabled for web browser but enabled for all command line activities such as curl on the computer.

We are not using Avast but here you can see how the feature works: https://blog.avast.com/2015/05/25/explaining-avasts-https-scanning-feature/

huckeduster commented 2 years ago

Ok thanks, that will not work in my environment. I had to install Docker Desktop meanwhile, where everything works out of the box. But in case someone comes up with a potential solution to my problem, I'd like to test it in order to switch back to colima.

eljohnson92 commented 2 years ago

it doesn't seem that colima is respecting host certs, such as /etc/docker/certs.d/*. Using Colima with a private registry gives me this

docker login <private_reg>
...
Error response from daemon: Get "<private_reg>": x509: certificate signed by unknown authority

This is with latest (as of today)

colima version
colima version HEAD-272db47
git commit: 272db4732b90390232ed9bdba955877f46a50552

runtime: docker
arch: x86_64
client: v20.10.12
server: v20.10.11

I think this issue is different than the top line issue, but based on only mounting ~ and /tmp/colima by default I don't think the host certs are being added into /etc/docker/certs.d. On Mac the location of these certs(based on what was used with Docker Desktop) was ~/.docker/certs.d.

I managed to use the following overrides.yaml with limactl directly but not with colima.

provision:
  - mode: user
     script: |
      #!/bin/bash
      sudo ln -sf /Users/${USER}/.docker/certs.d /etc/docker/certs.d
abiosoft commented 2 years ago

I managed to use the following overrides.yaml with limactl directly but not with colima.

provision:
  - mode: user
    script: |
    #!/bin/bash
    sudo ln -sf /Users/${USER}/.docker/certs.d /etc/docker/certs.d

If this is always the location /Users/${USER}/.docker/certs.d (regardless of the presence of Docker Desktop), then it can be implemented relatively easily.

eljohnson92 commented 2 years ago

All native linux docker instances will be looking for the /etc/docker/certs.d path, I'm guessing docker desktop decided to go with the ~/.docker/certs.d route so it didn't have to also mount files from the /etc/* path on the host and it could focus on files in the users home directory. Additionally, it looks like the docker CLI puts its config files under ~/.docker so I think most of the time this will be the case, but for users who have more homegrown docker-vm integration the certs.d path might live somewhere else.

huckeduster commented 2 years ago

https://docs.docker.com/desktop/mac/#add-tls-certificates => "Add client certificates" => ~/.docker/certs.d/ seems to be the right directory.

abiosoft commented 2 years ago

From https://docs.docker.com/desktop/mac/#add-client-certificates, I can see the following

When the Docker Desktop application starts, it copies the ~/.docker/certs.d folder on your Mac to the /etc/docker/certs.d directory on Moby (the Docker Desktop xhyve virtual machine).

I think that's settled, the behaviour can be replicated in Colima as well

abiosoft commented 2 years ago

can anyone try this and see if it fixes the docker cert issue? https://github.com/abiosoft/colima/releases/tag/v0.3.3-pre-02

ataraxus commented 2 years ago

@abiosoft thanks! this fixes exactly my issue!!! great cant wait to have it as a release in brew

eljohnson92 commented 2 years ago

this also fixes my issue, thanks!

bmichaud commented 2 years ago

I am having the same issue. I remove Docker Desktop entirely from my Intel MacBook prior to installing docker and Colima with Home-brew. I have

(NOTE: The message does not even list a valid URL to the image.)

It looks like the above fix was released in version 0.3.3 of Colima, and I have 0.3.4, so is this a new issue?

abiosoft commented 2 years ago

@bmichaud

(NOTE: The message does not even list a valid URL to the image.)

The output is truncated. Use --verbose flag if you do not want a truncated output.

Are you having problems downloading specifically on github.com? Will providing a mirror for the image improve the situation?

bmichaud commented 2 years ago

Thanks, @abiosoft, I noted that our firewall was blocking .iso files, so I downloaded the file and provided the image as a local location using the ~.lima/_config/override.yaml file. Now is just having trouble provisioning Docker.

[Ben.Michaud] [~/.lima/_config] $> colima start --cpu 4 --memory 8 --disk 60 --verbose INFO[0000] starting colima INFO[0000] starting ... context=vm

msg="Using the existing instance \"colima\"" msg="Attempting to download the image from \"~/.lima/_config/alpine-lima-clm-3.14.3-x86_64.iso\"" digest= msg="Downloaded image from \"~/.lima/_config/alpine-lima-clm-3.14.3-x8664.iso\"" msg="[hostagent] local user \"Ben.Michaud\" is not a valid Linux username (must match \"^[a-z][a-z0-9_-]*$\"); using \"lima\" username instead" msg="[hostagent] Starting QEMU (hint: to watch the boot progress, see \"/Users/Ben.Michaud/.lima/colima/serial.log\")" msg="SSH Local Port: 51636" msg="[hostagent] Waiting for the essential requirement 1 of 5: \"ssh\"" msg="[hostagent] The essential requirement 1 of 5 is satisfied" msg="[hostagent] Waiting for the essential requirement 2 of 5: \"user session is ready for ssh\"" msg="[hostagent] The essential requirement 2 of 5 is satisfied" msg="[hostagent] Waiting for the essential requirement 3 of 5: \"sshfs binary to be installed\"" msg="[hostagent] The essential requirement 3 of 5 is satisfied" msg="[hostagent] Waiting for the essential requirement 4 of 5: \"/etc/fuse.conf to contain \\"user_allow_other\\"\"" msg="[hostagent] The essential requirement 4 of 5 is satisfied" msg="[hostagent] Waiting for the essential requirement 5 of 5: \"the guest agent to be running\"" msg="[hostagent] The essential requirement 5 of 5 is satisfied" msg="[hostagent] Mounting \"/Users/Ben.Michaud\"" msg="[hostagent] Mounting \"/tmp/colima\"" msg="[hostagent] Waiting for the final requirement 1 of 1: \"boot scripts must have finished\"" msg="[hostagent] Forwarding \"/var/run/docker.sock\" (guest) to \"/Users/Ben.Michaud/.colima/docker.sock\" (host)" msg="[hostagent] The final requirement 1 of 1 is satisfied" msg="READY. Run limactl shell colima to open the shell." INFO[0026] provisioning ... context=docker INFO[0026] restarting VM to complete setup ... context=docker FATA[0026] error provisioning docker: error at 'restarting VM to complete setup': cannot restart, VM not previously started

abiosoft commented 2 years ago

@bmichaud delete and recreate it, should be fine.

colima delete
colima start
bmichaud commented 2 years ago

Thanks! Colima started without errors, but docker is not. I wasn't sure if Colima was going to start the docker daemon or not.

Cannot connect to the Docker daemon at unix:///Users/Ben.Michaud/.colima/docker.sock. Is the docker daemon running?

bmichaud commented 2 years ago

I have: colima version 0.3.4 git commit: 5a4a70481ca8d1e794677f22524e3c1b79a9b4ae

runtime: docker arch: x86_64 client: v20.10.13 server: v20.10.11

I'm working on a MacBook Pro in a Jamf/Zscaler environment. People are able to download images from docker hub in my environment if they use Docker Desktop.

I have placed the root certificates in my ~/.docker/certs.d directory and restarted colima, but I am still getting this error when I try to build a Docker image that has an alpine base image. It fails on the download of the base image:

[+] Building 0.5s (4/4) FINISHED
 => [internal] load build definition from Dockerfile                                                                         0.0s
 => => transferring dockerfile: 37B                                                                                          0.0s
 => [internal] load .dockerignore                                                                                            0.0s
 => => transferring context: 34B                                                                                             0.0s
 => CANCELED [internal] load metadata for docker.io/library/nginx:alpine                                                     0.4s
 => ERROR [internal] load metadata for docker.io/library/node:12.18.1-alpine3.12                                             0.4s
------
 > [internal] load metadata for docker.io/library/node:12.18.1-alpine3.12:
------
failed to solve with frontend dockerfile.v0: failed to create LLB definition: failed to do request: Head "https://registry-1.docker.io/v2/library/node/manifests/12.18.1-alpine3.12": x509: certificate signed by unknown authority

Is there a particular root CA certificate format I need? Does the file name need to be ca.crt? Does the format need to be anything in particular? I have tried a cert exported from Keychain Access in .cer format and in PEM format.

Perhaps I need to find intermediate certificates? I'm not sure where those would be.

bmichaud commented 2 years ago

I have tried adding a ~/.docker/daemon.json with an entry like this: "insecure-registries": ["registry-1.docker.io"], but this did not work.

I also triedexporting the root certs in .cer and PEM format and putting them into the ~/.docker/certs.d/ directory and restarting and even deleting colima and starting, but that did not work. Perhaps I am not putting the correct certs in the certs.d folder? Perhaps not in the right format?

Do I need to add something to the override.yaml to make this work? I tried adding this to the override.yaml as seen above:

provision:
  - mode: user
    script: |
    #!/bin/bash
    sudo ln -sf /Users/${USER}/.docker/certs.d /etc/docker/certs.d

But this resulted in a colima stop/start error:

FATA[0000] yaml: line 13: could not find expected ':'
WARN[0000] error detecting Lima config directory: error retrieving lima info: exit status 1
WARN[0000] falling back to default '$HOME/.lima'
TG-KK commented 2 years ago

For those on Mac, I figured out the process. You've to just manually update the default template file by running colima template and then edit the docker: {} to

docker:
  insecure-registries:
    - myregistry.com:5000
    - host.docker.internal:5000
    - etc....
chefren commented 1 year ago

From https://docs.docker.com/desktop/mac/#add-client-certificates, I can see the following

When the Docker Desktop application starts, it copies the ~/.docker/certs.d folder on your Mac to the /etc/docker/certs.d directory on Moby (the Docker Desktop xhyve virtual machine).

I think that's settled, the behaviour can be replicated in Colima as well

Previous link was moved: https://docs.docker.com/desktop/faqs/macfaqs/#add-client-certificates

AminMkh commented 1 year ago

For those on Mac, I figured out the process. You've to just manually update the default template file by running colima template and then edit the docker: {} to

docker:
  insecure-registries:
    - myregistry.com:5000
    - host.docker.internal:5000
    - etc....

Thank you, this worked, I am wondering however if I can use cli for configurations overrides, I noticed there is a --env option, but not documented or no examples found, I tried: colima start --cpu 4 --memory 8 --disk 100 --kubernetes -p xyz_profile --env docker=[insecure-registries:[repo.com]] --edit --editor codium and i realized it only added a env section at the bottom of the template:

# Environment variables for the virtual machine.
#
# EXAMPLE
# env:
#   KEY: value
#   ANOTHER_KEY: another value
#
# Default: {}
env:
  docker: '[insecure-registries:[repo.com]]'
abiosoft commented 1 year ago

I am wondering however if I can use cli for configurations overrides, I noticed there is a --env option, but not documented or no examples found

At the moment, it can only be configured via the config file.

joaoheytor commented 1 year ago

Hey, sorry to reopen this... I followed the steps to login into private ACRs but it seems it doesn't work for Kubernetes... any suggestions?

sky-philipalmeida commented 1 year ago

Hi, I went with the unsecure approach but maybe this pages help:

Self signed: https://docs.docker.com/registry/insecure/#use-self-signed-certificates

Unsecure: https://docs.docker.com/registry/insecure/#deploy-a-plain-http-registry

Every change using colima template requires a colima stop, colima start. You can check the status of the colima instance by executing limactl shell colima and then:

colima:/Users/user$ ls /etc/docker/certs.d/
colima:/Users/user$ cat /etc/docker/daemon.json
{
  "exec-opts": [
    "native.cgroupdriver=cgroupfs"
  ],
  "features": {
    "buildkit": true
  },
  "host-gateway-ip": "192.168.5.2",
  "insecure-registries": [
    "https://registry.xxx:443",
    "https://registry.xxx"
  ]
}
AzyCrw4282 commented 1 year ago

on a similar issue, i had been encountering issues with my new proxy. To fix it, i did the following:

have colima running 
run in a terminal:
limactl shell colima
sudo vi /etc/ssl/certs/ca-certificates.crt
append the <proxy> pem certificate to the end of the file   
save it 
exit shell
restart colima
DavidEdell commented 11 months ago

I had the same issue and was able to resolve it cleanly with a variation of the above for colima.

mkdir -p ~/.docker/certs.d
cp /PATH/TO/YOUR/CORP_OR_AV.cer ~/.docker/certs.d/CORP_OR_AV.crt
 # NOTE: If your CA was provided as a .cer file, be sure to rename it as .crt when copying it over.
colima stop
colima start
haspre commented 11 months ago

so, i followed quite a lot of the above workarounds... wasted like 2 hours on this.

nothing worked for me :(

I'm on mac 14.1.1 newest colima

I'll go install docker desktop again, some people just have to work.

kleinjoshuaa commented 10 months ago

Was not able to resolve this with any of the above workarounds

marquiswang commented 10 months ago

The workarounds worked on 0.5.6, but don't appear to work on 0.6.x.

jorijn commented 9 months ago

Watching. Isn't working for me either.

weikangchia commented 9 months ago

Facing the same issue also for colina 0.6.x, does anyone have any solution?

rtruong commented 9 months ago

Can confirm that the above workarounds no longer work on 0.6.x. Had to downgrade to 0.5.6.

momo182 commented 9 months ago

here's what i did to have it fixed:

colima ssh
sudo -s
cd /etc/ssl/certs
openssl s_client -showcerts -connect some.local.repo:8080 < /dev/null | awk '/-----BEGIN CERTIFICATE-----/ {p=1}; p; /-----END CERTIFICATE-----/ {p=0}' > ./some.local.repo.pem
update-ca-certificates --fresh
systemctl restart docker
k-ta-yamada commented 8 months ago

I was unable to docker pull, but the following steps solved the problem. Maybe this will help.

The cause was VPN software, but the situation was the same as the Avast case described in this comment.

Fortunately, I was able to get the cert file and used it (hereafter referred to as xxx.crt).

# Assume the working directory is the directory where xxx.crt exists.

colima ssh

# Copy xxx.crt from the host.
sudo cp xxx.crt /usr/local/share/ca-certificates/

# Apply
sudo update-ca-certificates

# Check: should hit 2 lines
ls -la /etc/ssl/certs/ | grep xxx.crt

# docker restart or
exit

# colima restart
colima restart

# Make sure you can docker pull
docker pull hello-world

I think you are basically doing the same thing with your comment here.

russianfool commented 8 months ago

This is what worked for me, mostly based on @momo182, to automatically pull the mitm certificate and trust it. The reason for doing it this way is:

  1. I'm not actually sure where the certificate resides on my operating system (might be just in the keychain?)
  2. If you run any kind of docker build commands that include package management, you need to re-run these commands inside the container itself too before using the package manager or wget, since container needs to trust these too.
  3. Supports multiple certificates in the chain (should only have to trust root, but idk).

Initially, docker pull hello-world fails with tls error error pulling image configuration: download failed after attempts=6: tls: failed to verify certificate: x509: certificate signed by unknown authority. You can plug in this one-liner to trust whatever certs https registry.hub.docker.com serves automatically:

colima ssh -- sudo sh -c 'cd /etc/ssl/certs && openssl s_client -showcerts -connect registry.hub.docker.com:443 < /dev/null | awk "/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/ {if (/-----BEGIN CERTIFICATE-----/) {if (out) close(out); out=\"certificate_mitm_\" ++n \".pem\"}; print > out}" && update-ca-certificates --fresh' && colima restart

Now, you can test and docker pull hello-world should succeed. You'll have to stick something similar in your Dockerfile, here it is not as a one-liner for readability (and might be different depending on base image):

cd /etc/ssl/certs
openssl s_client -showcerts -connect registry.hub.docker.com:443 < /dev/null | awk '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/ {if (/-----BEGIN CERTIFICATE-----/) {if (out) close(out); out="certificate_mitm_" ++n ".pem"}; print > out}'
update-ca-certificates --fresh

Of course, the base Ubuntu and other images don't always have the tools installed, but to install them using apt-get or dnf you'll need to pull them from secure repos. You can probably inject certs or tools from the host or do it manually, but can just pull the tools usually ignoring insecure certificates.

RUN apt-get update && apt-get install -y -o Acquire::https::Verify-Peer=false openssl ca-certificates
WandersonAlves commented 8 months ago

Hey @russianfool I think your comment should be added to the README.md. Ty and @momo182, I have been struggling with this for some time. Now it's solved. Thanks again!

ricardozanini commented 4 months ago

I had the same issue and was able to resolve it cleanly with a variation of the above for colima.

mkdir -p ~/.docker/certs.d
cp /PATH/TO/YOUR/CORP_OR_AV.cer ~/.docker/certs.d/CORP_OR_AV.crt
 # NOTE: If your CA was provided as a .cer file, be sure to rename it as .crt when copying it over.
colima stop
colima start

This worked for me instead of creating a dir named after the cert URI as stated in Docker docs.

hotwebmatter commented 3 months ago

I managed to use the following overrides.yaml with limactl directly but not with colima.

provision:
  - mode: user
    script: |
    #!/bin/bash
    sudo ln -sf /Users/${USER}/.docker/certs.d /etc/docker/certs.d

If this is always the location /Users/${USER}/.docker/certs.d (regardless of the presence of Docker Desktop), then it can be implemented relatively easily.

I tried this approach, but it did not run as my user; it ran as user lima. Therefore, it linked to the wrong directory:

$ ls -l /etc/docker/certs.d/
total 0
lrwxrwxrwx 1 root root 27 Jun 20 10:44 certs.d -> /Users/lima/.docker/certs.d
tks98 commented 1 month ago

Hello all, I am on macOS and created this script which resolved SSL issues for me. It exports the root certificates from the macOS Keychain, configures Colima to use these certificates, and restarts the Docker daemon. Sharing in case it proves helpful to someone else. 


#!/bin/bash

# Create directories
mkdir -p ~/.ca-certificates
mkdir -p ~/.colima/default

# Export certificates from Keychain
security find-certificate -a -p /System/Library/Keychains/SystemRootCertificates.keychain > ~/.ca-certificates/root_certs.pem
security find-certificate -a -p /Library/Keychains/System.keychain >> ~/.ca-certificates/root_certs.pem

# Create or update Colima configuration
cat << EOF > ~/.colima/default/colima.yaml
provision:
  - mode: system
    script: |
      CERTS="/Users/$(whoami)/.ca-certificates"
      cp \${CERTS}/* /usr/local/share/ca-certificates/
      update-ca-certificates
      systemctl daemon-reload
      systemctl restart docker
EOF

# Stop Colima if it's running
colima stop

# Start Colima with the new configuration
colima start

echo "Certificates exported and Colima configured and restarted"