[BUG] letsencrypt handshake not working on IPv6 only

chrismade commented 11 months ago

What happened?

I'm aware that the IPV6=yes feature is currently in beta and not ready for production - so this report is likely for the backlog. I created a simple static website - and tried to enable letsencrypt to acquire a certificate - however, that is failing - bunkerweb startup procedure stops at this point:

root-bw-scheduler-1  | [2023-11-25 10:07:55] - API - ℹ️  - Successfully sent API request to http://root-bunkerweb-1:5000/reload
root-bw-scheduler-1  | [2023-11-25 10:07:55] - SCHEDULER - ℹ️  - Successfully reloaded nginx
root-bw-scheduler-1  | [2023-11-25 10:07:55] - SCHEDULER - ℹ️  - Executing job scheduler ...
root-bunkerweb-1     | bwapi 10.20.30.3 - - [25/Nov/2023:10:07:55 +0000] "POST /reload HTTP/1.1" 200 58 "-" "bwapi"
root-bunkerweb-1     | 2023/11/25 10:07:55 [emerg] 50#50: cannot load certificate "/var/cache/bunkerweb/letsencrypt/etc/live/www.example.com/fullchain.pem": BIO_new_file() failed (SSL: error:80000002:system library::No such file or directory:calling fopen(/var/cache/bunkerweb/letsencrypt/etc/live/www.example.com/fullchain.pem, r) error:10000080:BIO routines::no such file)
root-bunkerweb-1     | 2023/11/25 10:08:00 [notice] 50#50: signal 17 (SIGCHLD) received from 115
root-bunkerweb-1     | 2023/11/25 10:08:00 [notice] 50#50: unknown process 51 exited with code 0

How to reproduce?

running bunkerweb v1.5.3 in docker on debian12 - made a simple static website under www.example.com for this case the connection is only working on IPv6 and hence I only made an AAAA DNS record the connectivity to bunkerweb on port 80 was tested from outside by curl and was ok then I changed the two letsencrypt lines in the config file from "no" to "yes" and restarted docker compose

letsencrypt's standard behavior is to use IPv6 / AAAA records first (which causes its own issues sometimes) so I assume it is sufficiently tested under IPv6 already

I was able to get a certificate successfully, so no "file not found" error anymore when I added a IPv4 portforwarding and an A record for the domain - I assume that there is just a tiny issue in the certificate challenge handshake to verify the domain URL path which does not yet work under IPv6 only

Configuration file(s) (yaml or .env)

version: "3.5"

services:

  bunkerweb:
    image: bunkerity/bunkerweb:1.5.3
    ports:
      - 80:8080
      - 443:8443
    labels:
      - "bunkerweb.INSTANCE=yes"
    environment:
      - DNS_RESOLVERS=8.8.8.8 8.8.4.4
      - USE_IPV6=yes
      - API_WHITELIST_IP=127.0.0.0/8 10.20.30.0/24
      - MULTISITE=yes
      - SERVER_NAME=www.example.com
      - www.example.com_SERVER_NAME=www.example.com
      - www.example.com_AUTO_LETS_ENCRYPT=yes
      - www.example.com_EMAIL_LETS_ENCRYPT=privacy@example.com
      - www.example.com_USE_LETS_ENCRYPT_STAGING=yes
    volumes:
      - www:/var/www/html
    networks:
      - bw-universe
      - bw-services

  bw-scheduler:
    image: bunkerity/bunkerweb-scheduler:1.5.3
    depends_on:
      - bunkerweb
      - bw-docker
    volumes:
      - bw-data:/data
    environment:
      - DOCKER_HOST=tcp://bw-docker:2375
    networks:
      - bw-universe
      - bw-docker

  bw-docker:
    image: tecnativa/docker-socket-proxy:nightly
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      - CONTAINERS=1
      - LOG_LEVEL=warning
    networks:
      - bw-docker

volumes:
  bw-data:
  www:

networks:
  bw-universe:
    name: bw-universe
    ipam:
      driver: default
      config:
        - subnet: 10.20.30.0/24
  bw-docker:
    name: bw-docker
  bw-services:
    name: bw-services
    enable_ipv6: true
    ipam:
      config:
        - subnet: fd00:13:37::/48
          gateway: fd00:13:37::1

Relevant log output

root-bw-scheduler-1  | [2023-11-25 10:07:55] - API - ℹ️  - Successfully sent API request to http://root-bunkerweb-1:5000/reload
root-bw-scheduler-1  | [2023-11-25 10:07:55] - SCHEDULER - ℹ️  - Successfully reloaded nginx
root-bw-scheduler-1  | [2023-11-25 10:07:55] - SCHEDULER - ℹ️  - Executing job scheduler ...
root-bunkerweb-1     | bwapi 10.20.30.3 - - [25/Nov/2023:10:07:55 +0000] "POST /reload HTTP/1.1" 200 58 "-" "bwapi"
root-bunkerweb-1     | 2023/11/25 10:07:55 [emerg] 50#50: cannot load certificate "/var/cache/bunkerweb/letsencrypt/etc/live/www.example.com/fullchain.pem": BIO_new_file() failed (SSL: error:80000002:system library::No such file or directory:calling fopen(/var/cache/bunkerweb/letsencrypt/etc/live/www.example.com/fullchain.pem, r) error:10000080:BIO routines::no such file)
root-bunkerweb-1     | 2023/11/25 10:08:00 [notice] 50#50: signal 17 (SIGCHLD) received from 115
root-bunkerweb-1     | 2023/11/25 10:08:00 [notice] 50#50: unknown process 51 exited with code 0

BunkerWeb version

1.5.3

What integration are you using?

Docker

Linux distribution (if applicable)

debian12

Removed private data

[X] I have removed all private data from the configuration file and the logs

Code of Conduct

[X] I agree to follow this project's Code of Conduct

TheophileDiot commented 11 months ago

Hi @chrismade, thank you for opening this issue. Let's encrypt will never generate a certificate for www.example.com as it is forbidden but Let's encrypt themself. try using a server_name that has a valid record and that you own and it should work just fine. Logs :

An unexpected error occurred:
Error creating new order :: Cannot issue for "www.example.com": The ACME server refuses to issue a certificate for this domain name, because it is forbidden by policy
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/bunkerweb/letsencrypt.log or re-run Certbot with -v for more details.

chrismade commented 11 months ago

@TheophileDiot thanks for pointing out that www.example.com is restricted by policy - and yes, I don't own this domain.

The essence of my bug report is that getting a letsencrypt certificate won't work for ANY domain if you only have an AAAA record and IPv6 connection (which it should) - only if you add an A record and IPv4 connection it will work.

bug reporting guideline request us to replace indivudual data by something some generic. So kindly replace www.example.com by any domain you own to reproduce the issue.

Can you pls have a look into this issue?

TheophileDiot commented 11 months ago

Oh alright, @chrismade. My bad. i'm currently investing this. I'll let you know if I find something.

bunkerity / bunkerweb