search

certbot / certbot-docker

94 stars 23 forks source link

The requested apache plugin does not appear to be installed #30

Closed kklepper closed 4 years ago

kklepper commented 4 years ago

I used https://github.com/wmnnd/nginx-certbot and expected this container to work fine out-of-the-box -- which it did, until renew was due and did not work. For a while I was just puzzled, but then I decided to investigate the issue.

Running certbot renew manually from inside the container I get

The requested nginx plugin does not appear to be installed

Trying certbot --nginx renew yields the same.

What to do?

Indeed, I do run certbot with nginx:

version: '3.7'

networks:
  proxy:
    external: true

services:
  n_proxy: 
    image: nginx:1.18.0-alpine
    hostname: n_proxy
    restart: on-failure:5
    networks:
      - proxy
    volumes:
      - /root/2proxy/nginx.conf:/etc/nginx/nginx.conf
      - /root/2proxy/nginx/log/:/var/log/nginx/
      - /root/2proxy/nginx/cache/:/etc/nginx/cache
      - /etc/letsencrypt/:/etc/letsencrypt/
    ports:
      - mode: host
        protocol: tcp
        published: 80
        target: 80
      - mode: host
        protocol: tcp
        published: 443
        target: 443
    command: "/bin/sh -c 'while :; do sleep 6h & wait $${!}; nginx -s reload; done & nginx -g \"daemon off;\"'"

  certbot:
    image: certbot/certbot
    restart: unless-stopped
    volumes:
      - /etc/letsencrypt:/etc/letsencrypt
      - /var/www/certbot:/var/www/certbot
    entrypoint: "/bin/sh -c 'trap exit TERM; while :; do certbot renew; sleep 12h & wait $${!}; done;'"

My certbot version is

/opt/certbot # certbot --version
certbot 1.5.0

Also, I cannot find out which operating system it runs on, so I have no idea which command for install would be correct:

/opt/certbot # uname -a
Linux 7e0a9a3dd132 4.18.0-147.8.1.el8_1.x86_64 #1 SMP Thu Apr 9 13:49:54 UTC 2020 x86_64 Linux

According to some hints I found googling, I should run

/opt/certbot # apt-get install python-certbot-apache -y
ash: apt-get: not found

so apt-get is not ok and apache does not make sense anyway. Looking at Dockerfile I deduce it is alpine, so I try 

/opt/certbot # apk add python-certbot-nginx
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/community/x86_64/APKINDEX.tar.gz
ERROR: unsatisfiable constraints:
  python-certbot-nginx (missing):
    required by: world[python-certbot-nginx]

Some more info:

/opt/certbot # env
QEMU_ARCH=x86_64
HOSTNAME=7e0a9a3dd132
PYTHON_PIP_VERSION=20.1.1
SHLVL=1
HOME=/root
GPG_KEY=E3FF2839C048B25C084DEBE9B26995E310250568
PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/eff16c878c7fd6b688b9b4c4267695cf1a0bf01b/get-pip.py
TERM=xterm
PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LANG=C.UTF-8
CERTBOT_VERSION=1.5.0
PYTHON_VERSION=3.8.3
PWD=/opt/certbot
PYTHON_GET_PIP_SHA256=b3153ec0cf7b7bbf9556932aa37e4981c35dc2a2c501d70d91d2795aa532be79

I ran out of ideas.

Your certificate (or certificates) for the names listed below will expire in 10 days (on 12 Aug 20 12:07 +0000).

kklepper commented 4 years ago

https://certbot.eff.org/docs/install.html

The Apache plugin currently requires an OS with augeas version 1.0; currently it supports modern OSes based on Debian, Ubuntu, Fedora, SUSE, Gentoo and Darwin.

I conclude that Apache plugin will not run on this container anyway. Right?

If so, how to get a renew working?

kklepper commented 4 years ago 
/opt/certbot # apk add python3-certbot-nginx
ERROR: unsatisfiable constraints:
  python3-certbot-nginx (missing):
    required by: world[python3-certbot-nginx]

/opt/certbot # apk add certbot-nginx
(1/36) Installing python3 (3.8.2-r1)
(2/36) Installing py3-setuptools (42.0.2-r0)
(3/36) Installing py3-cparser (2.19-r4)
(4/36) Installing py3-cffi (1.13.2-r0)
(5/36) Installing py3-idna (2.8-r3)
(6/36) Installing py3-asn1crypto (1.2.0-r1)
(7/36) Installing py3-six (1.13.0-r0)
(8/36) Installing py3-cryptography (2.8-r1)
(9/36) Installing py3-openssl (19.1.0-r0)
(10/36) Installing py3-josepy (1.2.0-r3)
(11/36) Installing py3-pbr (5.4.4-r0)
(12/36) Installing py3-mock (2.0.0-r6)
(13/36) Installing py3-tz (2019.3-r2)
(14/36) Installing py3-pyrfc3339 (1.1-r3)
(15/36) Installing py3-chardet (3.0.4-r3)
(16/36) Installing py3-certifi (2019.9.11-r2)
(17/36) Installing py3-urllib3 (1.25.7-r1)
(18/36) Installing py3-requests (2.22.0-r0)
(19/36) Installing py3-requests-toolbelt (0.9.1-r1)
(20/36) Installing py3-acme (1.0.0-r0)
(21/36) Installing py3-configargparse (0.15.2-r0)
(22/36) Installing py3-configobj (5.0.6-r7)
(23/36) Installing py3-distro (1.4.0-r3)
(24/36) Installing py3-distutils-extra (2.42-r1)
(25/36) Installing py3-future (0.18.2-r0)
(26/36) Installing py3-parsedatetime (2.5-r0)
(27/36) Installing py3-zope-interface (4.7.1-r0)
(28/36) Installing py3-zope-proxy (4.3.3-r0)
(29/36) Installing py3-zope-deferredimport (4.3.1-r2)
(30/36) Installing py3-zope-deprecation (4.4.0-r3)
(31/36) Installing py3-zope-event (4.4-r4)
(32/36) Installing py3-zope-hookable (5.0.0-r0)
(33/36) Installing py3-zope-component (4.6-r0)
(34/36) Installing certbot (1.0.0-r0)
(35/36) Installing py3-parsing (2.4.5-r1)
(36/36) Installing certbot-nginx (1.0.0-r0)
Executing busybox-1.31.1-r9.trigger
OK: 108 MiB in 75 packages

As far as I understood, this nginx should be listed in plugins.

/opt/certbot # certbot plugins
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
* standalone
Description: Spin up a temporary webserver
Interfaces: IAuthenticator, IPlugin
Entry point: standalone = certbot._internal.plugins.standalone:Authenticator

* webroot
Description: Place files in webroot directory
Interfaces: IAuthenticator, IPlugin
Entry point: webroot = certbot._internal.plugins.webroot:Authenticator
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

renew produces the same error.


/opt/certbot #  pip3 install certbot-nginx
Collecting certbot-nginx
  Downloading https://files.pythonhosted.org/packages/99/45/0cd67591e05edaec30270fe4c4f48b1da5448666de0787500bb69bde01ef/certbot_nginx-1.7.0-py2.py3-none-any.whl (45kB)
    100% |████████████████████████████████| 51kB 3.7MB/s
Requirement already satisfied: setuptools in /usr/local/lib/python3.8/site-packages (from certbot-nginx)
Requirement already satisfied: acme>=1.4.0 in ./src/acme (from certbot-nginx)
Requirement already satisfied: PyOpenSSL in /usr/local/lib/python3.8/site-packages (from certbot-nginx)
Requirement already satisfied: zope.interface in /usr/local/lib/python3.8/site-packages (from certbot-nginx)
Requirement already satisfied: pyparsing>=1.5.5 in /usr/local/lib/python3.8/site-packages (from certbot-nginx)
Collecting certbot>=1.6.0 (from certbot-nginx)
  Downloading https://files.pythonhosted.org/packages/b9/5d/bc8f1b87c9aca563c9a28ba253eecaaa73ebb3dae111286ed9ae95e61b80/certbot-1.7.0-py2.py3-none-any.whl (239kB)
    100% |████████████████████████████████| 245kB 3.2MB/s
Requirement already satisfied: cryptography>=1.2.3 in /usr/local/lib/python3.8/site-packages (from acme>=1.4.0->certbot-nginx)
Requirement already satisfied: josepy>=1.1.0 in /usr/local/lib/python3.8/site-packages (from acme>=1.4.0->certbot-nginx)
Requirement already satisfied: pyrfc3339 in /usr/local/lib/python3.8/site-packages (from acme>=1.4.0->certbot-nginx)
Requirement already satisfied: pytz in /usr/local/lib/python3.8/site-packages (from acme>=1.4.0->certbot-nginx)
Requirement already satisfied: requests[security]>=2.6.0 in /usr/local/lib/python3.8/site-packages (from acme>=1.4.0->certbot-nginx)
Requirement already satisfied: requests-toolbelt>=0.3.0 in /usr/local/lib/python3.8/site-packages (from acme>=1.4.0->certbot-nginx)
Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.8/site-packages (from acme>=1.4.0->certbot-nginx)
Requirement already satisfied: distro>=1.0.1 in /usr/local/lib/python3.8/site-packages (from certbot>=1.6.0->certbot-nginx)
Requirement already satisfied: ConfigArgParse>=0.9.3 in /usr/local/lib/python3.8/site-packages (from certbot>=1.6.0->certbot-nginx)
Requirement already satisfied: configobj in /usr/local/lib/python3.8/site-packages (from certbot>=1.6.0->certbot-nginx)
Requirement already satisfied: parsedatetime>=1.3 in /usr/local/lib/python3.8/site-packages (from certbot>=1.6.0->certbot-nginx)
Requirement already satisfied: zope.component in /usr/local/lib/python3.8/site-packages (from certbot>=1.6.0->certbot-nginx)
Requirement already satisfied: cffi!=1.11.3,>=1.8 in /usr/local/lib/python3.8/site-packages (from cryptography>=1.2.3->acme>=1.4.0->certbot-nginx)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/site-packages (from requests[security]>=2.6.0->acme>=1.4.0->certbot-nginx)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.8/site-packages (from requests[security]>=2.6.0->acme>=1.4.0->certbot-nginx)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.8/site-packages (from requests[security]>=2.6.0->acme>=1.4.0->certbot-nginx)
Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.8/site-packages (from requests[security]>=2.6.0->acme>=1.4.0->certbot-nginx)
Requirement already satisfied: zope.deferredimport>=4.2.1 in /usr/local/lib/python3.8/site-packages (from zope.component->certbot>=1.6.0->certbot-nginx)
Requirement already satisfied: zope.event in /usr/local/lib/python3.8/site-packages (from zope.component->certbot>=1.6.0->certbot-nginx)
Requirement already satisfied: zope.hookable>=4.2.0 in /usr/local/lib/python3.8/site-packages (from zope.component->certbot>=1.6.0->certbot-nginx)
Requirement already satisfied: zope.deprecation>=4.3.0 in /usr/local/lib/python3.8/site-packages (from zope.component->certbot>=1.6.0->certbot-nginx)
Requirement already satisfied: pycparser in /usr/local/lib/python3.8/site-packages (from cffi!=1.11.3,>=1.8->cryptography>=1.2.3->acme>=1.4.0->certbot-nginx)
Requirement already satisfied: zope.proxy in /usr/local/lib/python3.8/site-packages (from zope.deferredimport>=4.2.1->zope.component->certbot>=1.6.0->certbot-nginx)
Installing collected packages: certbot, certbot-nginx
  Found existing installation: certbot 1.5.0
    Uninstalling certbot-1.5.0:
      Successfully uninstalled certbot-1.5.0
Successfully installed certbot-1.7.0 certbot-nginx-1.7.0
You are using pip version 9.0.1, however version 20.2.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Ok, try again:


/opt/certbot # certbot plugins
An unexpected error occurred:
pkg_resources.VersionConflict: (acme 1.5.0 (/opt/certbot/src/acme), Requirement.parse('acme>=1.6.0'))
Please see the logfile '/tmp/tmppkox5gdv/log' for more details.

Will try to fix that. Have to take a break now.

kklepper commented 4 years ago

Next run.

/opt/certbot # pip install --no-cache-dir --no-deps \
>         --editable src/acme \
>         --editable src/certbot
Obtaining file:///opt/certbot/src/acme
Obtaining file:///opt/certbot/src/certbot
Installing collected packages: acme, certbot
  Attempting uninstall: acme
    Found existing installation: acme 1.5.0
    Uninstalling acme-1.5.0:
      Successfully uninstalled acme-1.5.0
  Running setup.py develop for acme
  Attempting uninstall: certbot
    Found existing installation: certbot 1.7.0
    Uninstalling certbot-1.7.0:
      Successfully uninstalled certbot-1.7.0
  Running setup.py develop for certbot
Successfully installed acme certbot

/opt/certbot # certbot renew --dry-run
An unexpected error occurred:
pkg_resources.VersionConflict: (certbot 1.5.0 (/opt/certbot/src/certbot), Requirement.parse('certbot>=1.6.0'))
Please see the logfile '/tmp/tmpryh2tnrm/log' for more details.

Ok then. git clone https://github.com/certbot-docker/certbot-docker.git and ./build.sh v0.35.0

You are using pip version 9.0.1, however version 20.2.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Directory 'src/certbot' is not installable. File 'setup.py' not found.
You are using pip version 9.0.1, however version 20.2.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
The command '/bin/sh -c apk add --no-cache --virtual .build-deps         gcc         linux-headers         openssl-dev         musl-dev         libffi-dev     && python pipstrap.py     && pip install -r dependency-requirements.txt     && pip install --no-cache-dir --no-deps         --editable src/acme         --editable src/certbot && apk del .build-deps' returned a non-zero code: 1

Well, that's not nice, is it? Looking at ./build.sh, I notice

CERTBOT_PLUGINS_DOCKER_REPOS=(
    "certbot/dns-dnsmadeeasy"
    "certbot/dns-dnsimple"
    "certbot/dns-ovh"
    "certbot/dns-cloudflare"
    "certbot/dns-cloudxns"
    "certbot/dns-digitalocean"
    "certbot/dns-google"
    "certbot/dns-luadns"
    "certbot/dns-nsone"
    "certbot/dns-rfc2136"
    "certbot/dns-route53"
    "certbot/dns-gehirn"
    "certbot/dns-linode"
    "certbot/dns-sakuracloud"
)

No apache plugin, no nginx plugin mentioned here, they are hopefully in one of these repos. Which one to choose?

New approach. docker pull certbot/certbot:amd64-latest

certbot/certbot                                          amd64-latest        994e18b9d74d        14 hours ago        92.2MB
certbot/certbot                                          latest              ffd735f22ba5        2 months ago        128MB

Take certbot down, restart new version:

/opt/certbot # certbot renew --dry-run
Another instance of Certbot is already running.

Well, let's see.

docker ps -a | grep "cert"
e9878e5479d8        certbot/certbot:amd64-latest                                    "/bin/sh -c 'trap ex…"   2 minutes ago       Up 2 minutes                80/tcp, 443/tcp                                                                                                                    2proxy_certbot_1

What is happening here?

kklepper commented 4 years ago

So it is not a container. ps aux | grep "[c]ert"

root      62386  0.0  0.0   1576     4 ?        Ss   11:35   0:00 /bin/sh -c trap exit TERM; while :; do certbot renew; sleep 12h & wait ${!}; done;

Looks like the leftover from my build.sh attempt. kill 1762 should do. certbot renew --dry-run. 

Could not choose appropriate plugin: The requested apache plugin does not appear to be installed

Full circle.

kklepper commented 4 years ago 
$ apk add certbot-nginx

$ pip3 install certbot-nginx

$ certbot plugins
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
* nginx
Description: Nginx Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: nginx = certbot_nginx._internal.configurator:NginxConfigurator

* standalone
Description: Spin up a temporary webserver
Interfaces: IAuthenticator, IPlugin
Entry point: standalone = certbot._internal.plugins.standalone:Authenticator

* webroot
Description: Place files in webroot directory
Interfaces: IAuthenticator, IPlugin
Entry point: webroot = certbot._internal.plugins.webroot:Authenticator
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Finally. certbot renew --dry-run. Again The requested apache plugin does not appear to be installed..

Now a bold conjecture: certbot renew --dry-run --nginx.

Cert is due for renewal, auto-renewing...
Could not choose appropriate plugin: The nginx plugin is not working; there may be problems with your existing configuration.
The error was: NoInstallationError("Could not find a usable 'nginx' binary. Ensure nginx exists, the binary is executable, and your PATH is set correctly.")

Huh? nginx is running in a container:


networks:
  proxy:
    external: true

services:
  n_proxy: 
    image: nginx:1.18.0-alpine
    hostname: n_proxy
    restart: on-failure:5
    networks:
      - proxy
    volumes:
      - /root/2proxy/nginx.conf:/etc/nginx/nginx.conf
      - /root/2proxy/nginx/log/:/var/log/nginx/
      - /root/2proxy/nginx/cache/:/etc/nginx/cache
      - /etc/letsencrypt/:/etc/letsencrypt/
    ports:
      - mode: host
        protocol: tcp
        published: 80
        target: 80
      - mode: host
        protocol: tcp
        published: 443
        target: 443
    command: "/bin/sh -c 'while :; do sleep 6h & wait $${!}; nginx -s reload; done & nginx -g \"daemon off;\"'"

  certbot:
#    image: certbot/certbot
    image: certbot/certbot:amd64-latest
    restart: unless-stopped
    volumes:
      - /etc/letsencrypt:/etc/letsencrypt
      - /var/www/certbot:/var/www/certbot
    entrypoint: "/bin/sh -c 'trap exit TERM; while :; do certbot renew; sleep 12h & wait $${!}; done;'"
kklepper commented 4 years ago

and your PATH is set correctly.

Hm. This does not refer to host? Or does it? Incidentally, I do have nginx on the host as well.

$ which nginx
/usr/sbin/nginx

No, this cannot be. We are in a container. So does certbot expect nginx to live in the same container? I guess no. Both are in the same network, so why cannot certbot find nginx?

kklepper commented 4 years ago

Also, I could create the certificates, so why can't I renew?

kklepper commented 4 years ago

Why can't certbot see nginx?

ping: bad address 'nginx'
/opt/certbot $ apk add ping
ERROR: unsatisfiable constraints:
  ping (missing):
    required by: world[ping]
/opt/certbot $ apk add iputils
(1/2) Installing libcap (2.27-r0)
(2/2) Installing iputils (20190709-r0)
Executing busybox-1.31.1-r16.trigger
OK: 96 MiB in 81 packages
/opt/certbot $ ping nginx
ping: nginx: Name does not resolve

Hm. It should.

Oh, my configuration is faulty, I guess. certbot is not on the network. I have to add an internal network nc and change the name of the nginx container to nginx.

version: '3.7'

networks:
  nc:  
  proxy:
    external: true

services:
  nginx: 
#  n_proxy: 
    image: nginx:1.18.0-alpine
    hostname: nginx
#    hostname: n_proxy
    restart: on-failure:5
    networks:
      - proxy
      - nc
    volumes:
      - /root/2proxy/nginx.conf:/etc/nginx/nginx.conf
      - /root/2proxy/nginx/log/:/var/log/nginx/
      - /root/2proxy/nginx/cache/:/etc/nginx/cache
      - /etc/letsencrypt/:/etc/letsencrypt/
    ports:
      - mode: host
        protocol: tcp
        published: 80
        target: 80
      - mode: host
        protocol: tcp
        published: 443
        target: 443
    command: "/bin/sh -c 'while :; do sleep 6h & wait $${!}; nginx -s reload; done & nginx -g \"daemon off;\"'"

  certbot:
#    image: certbot/certbot
    image: certbot/certbot:amd64-latest
    restart: unless-stopped
    networks:
      - nc
    volumes:
      - /etc/letsencrypt:/etc/letsencrypt
      - /var/www/certbot:/var/www/certbot
    entrypoint: "/bin/sh -c 'trap exit TERM; while :; do certbot renew; sleep 12h & wait $${!}; done;'"

and stop and restart the setting:

$ docker-compose down --remove-orphans; docker-compose up -d
$ id=$(docker ps -a  | grep "certbot" | grep -v "xited" | awk '{print $1}') && docker exec -it $id sh
/opt/certbot # ping nginx
PING nginx (172.28.0.3): 56 data bytes
64 bytes from 172.28.0.3: seq=0 ttl=64 time=0.114 ms
64 bytes from 172.28.0.3: seq=1 ttl=64 time=0.110 ms
64 bytes from 172.28.0.3: seq=2 ttl=64 time=0.109 ms
64 bytes from 172.28.0.3: seq=3 ttl=64 time=0.131 ms
64 bytes from 172.28.0.3: seq=4 ttl=64 time=0.107 ms
64 bytes from 172.28.0.3: seq=5 ttl=64 time=0.107 ms
^C
--- nginx ping statistics ---
6 packets transmitted, 6 packets received, 0% packet loss
round-trip min/avg/max = 0.107/0.113/0.131 ms

So far so good. certbot can see nginx, the plugin is installed:

/opt/certbot # certbot plugins
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
* nginx
Description: Nginx Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: nginx = certbot_nginx._internal.configurator:NginxConfigurator

* standalone
Description: Spin up a temporary webserver
Interfaces: IAuthenticator, IPlugin
Entry point: standalone = certbot._internal.plugins.standalone:Authenticator

* webroot
Description: Place files in webroot directory
Interfaces: IAuthenticator, IPlugin
Entry point: webroot = certbot._internal.plugins.webroot:Authenticator
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

But it does not solve the problem:

/opt/certbot # certbot renew --dry-run --nginx
[...]
Could not choose appropriate plugin: The nginx plugin is not working; there may be problems with your existing configuration.
The error was: NoInstallationError("Could not find a usable 'nginx' binary. Ensure nginx exists, the binary is executable, and your PATH is set correctly.")

Oh my. Quite a story. Nobody here to give a hint?

AvverbioPronome commented 4 years ago

Oh my. Quite a story. Nobody here to give a hint?

There is no nginx inside that container, you can't use the nginx plugin. You should use standalone or webroot (or another acme client).

kklepper commented 4 years ago

@AvverbioPronome

Thank you very much. This was it.

It turned out that I was on the wrong track. The apache message was misleading and a residue of my approach to get a hold on this topic.

To quote from https://community.letsencrypt.org/t/the-nginx-plugin-is-not-working/130294/10:

Inspecting the logs, I reconstruct as follows:

  • I first worked with the official tutorial involving Apache
  • to this end I had to fire up httpd standalone
  • this would not work because my proxy runs on port 80
  • so I had to shut down my proxy
  • I obtained a certificate

Now this cannot work in production, so I looked for a docker solution and found Nginx and Let’s Encrypt with Docker in Less Than 5 Minutes, i.e. wmnnd. This is the boilerplate for my proxy.

@_az

Just calling certbot renew should use the webroot plugin automatically (as it remembers what was initially used), unless you changed it at some point.

This explains why it remembers Apache.

There were quite some more things to learn here. To get the rest of the story see in particular the last entry at the community issue mentioned above.