The 'allow insecure connections' flag (`-I`) does not apply to _most_ `apt` operations

kquinsland commented 2 months ago

Description of Issue/Question

Neither of the "insecure DL" options (-I and -l) are carried over to apt in most cases.

In corporate environments it is common to both a) have all traffic go through a MITM proxy and b) not be able to get the MITM root CA loaded in some environments.

Because of this, it's common to see errors like this:

$ apt-get update
<...>
       Ign:1 https://repo.saltproject.io/salt/py3/ubuntu/22.04/arm64/latest jammy InRelease
       Err:1 https://repo.saltproject.io/salt/py3/ubuntu/22.04/arm64/latest jammy InRelease
         Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown.  Could not handshake: Error in the certificate verification. [IP: a.b.c.d]
<...>

This is - unfortunately - expected. The man in the middle isn't trusted so apt - rightfully - refuses to pull.

The simplest way to work around this is by adding a -o Acquire::https::Verify-Peer=false to the apt ... command. Yes, I know that this can be put into the /etc/apt/config but when that happens, it applies to everything everywhere all the time which isn't making an already sub-optimal situation better.

In any case, the multiple apt ... calls in bootstrap.sh don't immediately or explicitly fail in this case; bootstrapping does not report errors.

Only after bootstrap is done and salt-call is run do I hit an error:

kitchen@80a9c3deb4e2:~$ sudo -E salt-call --state-output=changes --config-dir=/tmp/kitchen/etc/salt state.highstate --retcode-passthrough
<...>
[ERROR   ] Failed to import module pip, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 766, in _load_module
    mod = self.run(spec.loader.load_module)
  File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1201, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1216, in _run_as
    return _func_or_method(*args, **kwargs)
  File "<frozen importlib._bootstrap_external>", line 548, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 1063, in load_module
  File "<frozen importlib._bootstrap_external>", line 888, in load_module
  File "<frozen importlib._bootstrap>", line 290, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 719, in _load
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/lib/python3/dist-packages/salt/modules/pip.py", line 87, in <module>
    import pkg_resources  # pylint: disable=3rd-party-module-not-gated
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3267, in <module>
    def _initialize_master_working_set():
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3241, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3279, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 573, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 891, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 777, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'contextvars' distribution was not found and is required by salt

While not ideal, a manual pip install ... should work, right?

kitchen@80a9c3deb4e2:~$ which pip
kitchen@80a9c3deb4e2:~$ which pip3

Nope, because pip never got installed because apt skipped pulling packages for the https repos.

Setup

I am using the bootstrap.sh script with a ubuntu2204 container image. The specific args being fed are:

-l -I -X -P -D -p git -p nano stable

Steps to Reproduce Issue

I am using kitchen but the gist is ubuntu:2204 container image with salt-bootstrap.sh -l -I -X -P -D -p git -p nano stable being run on a host where all traffic goes through a MITM proxy.

Critically, the CA root for the MITM proxy is not loaded into the container environment so all https traffic appears to have a cert mis-match error.

Versions and Systems

kitchen@28eaf5100351:/tmp/kitchen$ ./bootstrap.sh -v
./bootstrap.sh -- Version 2024.07.23

kquinsland commented 2 months ago

I have a fix but it's currently only applied to the various apt calls. I am not currently in a good position to figure out the -o Acquire::https::Verify-Peer=false equivalent for the other package managers.

The .patch containing the fix is attached.

My question: Should I open a PR with just this fix or not bother until similar is implemented for the other package managers, too?

apt-insecure-everywhere.patch

dmurphy18 commented 1 month ago

@kquinsland I believe you are expecting more from these flags than is the original intention, from the script

 360     -I  If set, allow insecure connections while downloading any files. For
 361         example, pass '--no-check-certificate' to 'wget' or '--insecure' to
 362         'curl'. On Debian and Ubuntu, using this option with -U allows obtaining
 363         GnuPG archive keys insecurely if distro has changed release signatures.
.
.
 378     -l  Disable ssl checks. When passed, switches "https" calls to "http" where
 379         possible.

The -l merely alters https handling, as shown here

 825 # Check the _DISABLE_SSL value and set HTTP or HTTPS.
 826 if [ "$_DISABLE_SSL" -eq $BS_TRUE ]; then
 827     HTTP_VAL="http"
 828 else
 829     HTTP_VAL="https"
 830 fi

The -I is for handling communications insecurely, that is, curl, wget, etc.

 839 # Handle the insecure flags
 840 if [ "$_INSECURE_DL" -eq $BS_TRUE ]; then               
 841     _CURL_ARGS="${_CURL_ARGS} --insecure"
 842     _FETCH_ARGS="${_FETCH_ARGS} --no-verify-peer"
 843     _GPG_ARGS="${_GPG_ARGS} --keyserver-options no-check-cert"
 844     _WGET_ARGS="${_WGET_ARGS} --no-check-certificate"
 845 else
 846     _GPG_ARGS="${_GPG_ARGS} --keyserver-options ca-cert-file=/etc/ssl/certs/ca-certificates.crt"
 847 fi

In order to have it handle your special case of a MITM proxy, it might be better to provide a PR since this will provide a complete implementation (working example). Any PR submitted should also include tests which exercise the additional code.

saltstack / salt-bootstrap