search

hpc / charliecloud

Now hosted on GitLab.
https://gitlab.com/charliecloud/main
Apache License 2.0
312 stars 61 forks source link

ch-image pull: proxy error #998

Open ajyounge opened 3 years ago

ajyounge commented 3 years ago

Using Charlielcoud from master (0.23~pre+4715a0f), ch-image fails to properly pull an image on a system. Suspect problem is with Charliecloud not able to use basic PROXY setup and the corporate SSL certificate installe don host. I cannot pull a basic image from likely any external registry due to this bug. Tested on aarch64 but suspect problem with Charliecloud is independent of uarch.

Also tested against rootless Podman using the same Dockerfile, which properly detects and uses the system proxy settings and works as expected, verifying the Dockerifle.

$ env | grep -i http
http_proxy=http://user:nopass@blah.gov:80/
HTTPS_PROXY=http://user:nopass@blah.gov:80/
https_proxy=http://user:nopass@blah.gov:80/
HTTP_PROXY=http://user:nopass@blah.gov:80/
$ ch-image --version
0.23~pre+4715a0f
$ ch-image --dependencies
$ cat Dockerfile.centos7
FROM centos:7
RUN echo hello
RUN yum install -y openssh
$ ch-image build -f Dockerfile.centos7 .
  1 FROM centos:7
manifest: downloading
error: GET failed: HTTPSConnectionPool(host='registry-1.docker.io', port=443): Max retries exceeded with url: /v2/library/centos/manifests/7 (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))

$ podman build -f Dockerfile.centos7 .
STEP 1: FROM centos:7
STEP 2: RUN echo hello
hello
0d0d9678b941b168af0dffa1114685e2c2707f97a4de23f1b3e5b9429485ad86
STEP 3: RUN yum install -y openssh
Loaded plugins: fastestmirror, ovl
Determining fastest mirrors
 * base: ftp.osuosl.org
 * extras: ftp.osuosl.org
 * updates: ftp.osuosl.org
base                                                     | 3.6 kB     00:00
extras                                                   | 2.9 kB     00:00
updates                                                  | 2.9 kB     00:00
(1/4): base/7/aarch64/group_gz                             | 153 kB   00:00
(3/4): extras/7/aarch64/pr 7% [=                ]  0.0 B/s | 496 kB   --:-- ET(3/4): extras/7/aarch64/pr 16% [==-             ] 634 kB/s | 1.0 MB   00:08 ET(2/4): updates/7/aarch64/primary_db                        | 916 kB   00:00
(4/4): extras/7/aarch64/pr 30% [====-           ] 770 kB/s | 1.9 MB   00:05 ET(4/4): extras/7/aarch64/pr 35% [=====-          ] 784 kB/s | 2.2 MB   00:05 ET(3/4): base/7/aarch64/prim 40% [======-         ] 797 kB/s | 2.5 MB   00:04 ET(3/4): base/7/aarch64/prim 45% [=======         ] 808 kB/s | 2.8 MB   00:04 ET(3/4): base/7/aarch64/prim 50% [========        ] 820 kB/s | 3.1 MB   00:03 ET(3/4): base/7/aarch64/prim 56% [========-       ] 831 kB/s | 3.4 MB   00:03 ET(4/4): extras/7/aarch64/pr 61% [=========-      ] 840 kB/s | 3.7 MB   00:02 ET(4/4): extras/7/aarch64/pr 66% [==========-     ] 851 kB/s | 4.1 MB   00:02 ET(4/4): extras/7/aarch64/pr 71% [===========     ] 857 kB/s | 4.4 MB   00:02 ET(4/4): extras/7/aarch64/pr 76% [============    ] 865 kB/s | 4.7 MB   00:01 ET(3/4): base/7/aarch64/prim 81% [=============   ] 873 kB/s | 5.0 MB   00:01 ET(3/4): base/7/aarch64/prim 86% [=============-  ] 880 kB/s | 5.3 MB   00:00 ET(3/4): base/7/aarch64/prim 91% [==============- ] 887 kB/s | 5.6 MB   00:00 ET(3/4): base/7/aarch64/primary_db                           | 4.9 MB   00:05
(4/4): extras/7/aarch64/pr 96% [=============== ] 837 kB/s | 5.9 MB   00:00 ET(4/4): extras/7/aarch64/primary_db                         | 230 kB   00:05
Resolving Dependencies
--> Running transaction check
---> Package openssh.aarch64 0:7.4p1-21.el7 will be installed
--> Processing Dependency: libfipscheck.so.1()(64bit) for package: openssh-7.4p1-21.el7.aarch64
--> Running transaction check
---> Package fipscheck-lib.aarch64 0:1.4.1-6.el7 will be installed
--> Processing Dependency: /usr/bin/fipscheck for package: fipscheck-lib-1.4.1-6.el7.aarch64
--> Running transaction check
---> Package fipscheck.aarch64 0:1.4.1-6.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package               Arch            Version               Repository    Size
================================================================================
Installing:
 openssh               aarch64         7.4p1-21.el7          base         503 k
Installing for dependencies:
 fipscheck             aarch64         1.4.1-6.el7           base          22 k
 fipscheck-lib         aarch64         1.4.1-6.el7           base          10 k

Transaction Summary
================================================================================
Install  1 Package (+2 Dependent packages)

Total download size: 535 k
Installed size: 2.2 M
Downloading packages:
warning: /var/cache/yum/aarch64/7/base/packages/fipscheck-1.4.1-6.el7.aarch64.rpm: Header V4 RSA/SHA1 Signature, key ID 305d49d6: NOKEY
Public key for fipscheck-1.4.1-6.el7.aarch64.rpm is not installed
(1/3): fipscheck-1.4.1-6.el7.aarch64.rpm                   |  22 kB   00:00
(2/3): fipscheck-lib-1.4.1-6.el7.aarch64.rpm               |  10 kB   00:00
(3/3): openssh-7.4p1-21.el 47% [=======-        ]  0.0 B/s | 256 kB   --:-- ET(3/3): openssh-7.4p1-21.el 59% [=========-      ]  61 kB/s | 319 kB   00:03 ET(3/3): openssh-7.4p1-21.el7.aarch64.rpm                    | 503 kB   00:01
--------------------------------------------------------------------------------
Total                                              407 kB/s | 535 kB  00:01
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Importing GPG key 0xF4A80EB5:
 Userid     : "CentOS-7 Key (CentOS 7 Official Signing Key) <security@centos.org>"
 Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
 Package    : centos-release-7-9.2009.0.el7.centos.aarch64 (@instCentOS)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7-aarch64
Importing GPG key 0x305D49D6:
 Userid     : "CentOS AltArch SIG - AArch64 (http://wiki.centos.org/SpecialInterestGroup/AltArch/AArch64) <security@centos.org>"
 Fingerprint: ef8f 3ca6 6efd f32b 36cd adf7 6c7c b6ef 305d 49d6
 Package    : centos-release-7-9.2009.0.el7.centos.aarch64 (@instCentOS)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7-aarch64
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : fipscheck-lib-1.4.1-6.el7.aa [                             ] 1/  Installing : fipscheck-lib-1.4.1-6.el7.aa [###########################  ] 1/  Installing : fipscheck-lib-1.4.1-6.el7.aa [############################ ] 1/  Installing : fipscheck-lib-1.4.1-6.el7.aarch64                            1/3
  Installing : fipscheck-1.4.1-6.el7.aarch6 [                             ] 2/  Installing : fipscheck-1.4.1-6.el7.aarch6 [############                 ] 2/  Installing : fipscheck-1.4.1-6.el7.aarch6 [#############                ] 2/  Installing : fipscheck-1.4.1-6.el7.aarch6 [#########################    ] 2/  Installing : fipscheck-1.4.1-6.el7.aarch6 [##########################   ] 2/  Installing : fipscheck-1.4.1-6.el7.aarch6 [###########################  ] 2/  Installing : fipscheck-1.4.1-6.el7.aarch6 [############################ ] 2/  Installing : fipscheck-1.4.1-6.el7.aarch64                                2/3
  Installing : openssh-7.4p1-21.el7.aarch64 [                             ] 3/  Installing : openssh-7.4p1-21.el7.aarch64 [#                            ] 3/  Installing : openssh-7.4p1-21.el7.aarch64 [##                           ] 3/  Installing : openssh-7.4p1-21.el7.aarch64 [###                          ] 3/  Installing : openssh-7.4p1-21.el7.aarch64 [####                         ] 3/  Installing : openssh-7.4p1-21.el7.aarch64 [#####                        ] 3/  Installing : openssh-7.4p1-21.el7.aarch64 [######                       ] 3/  Installing : openssh-7.4p1-21.el7.aarch64 [#######                      ] 3/  Installing : openssh-7.4p1-21.el7.aarch64 [########                     ] 3/  Installing : openssh-7.4p1-21.el7.aarch64 [#########                    ] 3/  Installing : openssh-7.4p1-21.el7.aarch64 [##########                   ] 3/  Installing : openssh-7.4p1-21.el7.aarch64 [###########                  ] 3/  Installing : openssh-7.4p1-21.el7.aarch64 [############                 ] 3/  Installing : openssh-7.4p1-21.el7.aarch64 [#############                ] 3/  Installing : openssh-7.4p1-21.el7.aarch64 [##############               ] 3/  Installing : openssh-7.4p1-21.el7.aarch64 [###############              ] 3/  Installing : openssh-7.4p1-21.el7.aarch64 [################             ] 3/  Installing : openssh-7.4p1-21.el7.aarch64 [#################            ] 3/  Installing : openssh-7.4p1-21.el7.aarch64 [##################           ] 3/  Installing : openssh-7.4p1-21.el7.aarch64 [###################          ] 3/  Installing : openssh-7.4p1-21.el7.aarch64 [####################         ] 3/  Installing : openssh-7.4p1-21.el7.aarch64 [#####################        ] 3/  Installing : openssh-7.4p1-21.el7.aarch64 [######################       ] 3/  Installing : openssh-7.4p1-21.el7.aarch64 [##########################   ] 3/  Installing : openssh-7.4p1-21.el7.aarch64 [###########################  ] 3/  Installing : openssh-7.4p1-21.el7.aarch64 [############################ ] 3/  Installing : openssh-7.4p1-21.el7.aarch64                                 3/3
  Verifying  : fipscheck-1.4.1-6.el7.aarch64                                1/3
  Verifying  : fipscheck-lib-1.4.1-6.el7.aarch64                            2/3
  Verifying  : openssh-7.4p1-21.el7.aarch64                                 3/3

Installed:
  openssh.aarch64 0:7.4p1-21.el7

Dependency Installed:
  fipscheck.aarch64 0:1.4.1-6.el7      fipscheck-lib.aarch64 0:1.4.1-6.el7

Complete!
STEP 4: COMMIT
e7ada9f9672713066cab4248c1105b7f9f3c2c9b5ace387d9b71b3715561145d
reidpr commented 3 years ago

I can't reproduce this. I'm sure it depends on the details of your proxy setup. Can you point us to documentation on that?

ajyounge commented 3 years ago

Documentation on the proxy is basically in the example envars listed above, just with local URLs. There are also root certificates on the host machine that enable valid SSL/TLS traffic from the corporate network. I can share more details if you have access to Sandia's corporate network. While certainly annoying, this setup is fairly common for many enterprise networks.

Given the error from Charliecloud, I'm not sure which one (or both?) is causing the issue; proxy or certificates. I can only assume CH can be configured to pick up proxy and use the system certs like any normal process? However I am not aware of any way beyond standard system envars to configure Charliecloud.

reidpr commented 3 years ago

Does ch-image --tls-no-verify build -f Dockerfile.centos7 . work? This will turn off TLS certificate verification.

reidpr commented 3 years ago

I was able to reproduce this. Configuring the host system to trust the proxy’s root cert fixed the pull itself.

However, the image also needs to be configured to trust the relevant certificates if it does any TLS stuff.