search

nginx-proxy / acme-companion

Automated ACME SSL certificate generation for nginx-proxy
MIT License
7.39k stars 820 forks source link

curl endless loop with 3.x Kernels on version >= 2.2.9 #1084

Closed vertIcod3r closed 8 months ago

vertIcod3r commented 8 months ago

If your are using the latest image tag and recently updated your image: please make sure you've checked the required read on the project's README.

HTTPS does not work / certificate aren't created : please check in your acme-companion container logs if an authorization or verify error is mentioned, if it is please do the following before opening an issue:

Bug description

After the update to 2.2.9 certificate creation is no longer successful. The command: "curl -s --unix-socket /var/run/docker.sock -X GET http://localhost/containers/json" seems to hang on the container. Utilizing 100% cpu. The only output on the docker logs for the acme container is: "Info: running acme-companion version v2.2.9"

acme-companion image version

Info: running acme-companion version v2.2.9

If this first log line isn't present you are using a v1 image: please provide the tagged version you are using. If you are not using a tagged version latest, please try again with a tagged release before opening an issue (the last v1 tagged release is v1.13.1).

nginx-proxy's Docker configuration

Even with the basic docker compose from the docs:

version: '2'

services:
  nginx-proxy:
    image: nginxproxy/nginx-proxy
    container_name: nginx-proxy
    ports:
      - "6302:80"
      - "6303:443"
    volumes:
      - conf:/etc/nginx/conf.d
      - vhost:/etc/nginx/vhost.d
      - html:/usr/share/nginx/html
      - certs:/etc/nginx/certs:ro
      - /var/run/docker.sock:/tmp/docker.sock:ro
    network_mode: bridge

  acme-companion:
    image: nginxproxy/acme-companion:2.2.9
    container_name: nginx-proxy-acme
    environment:
      - DEFAULT_EMAIL=mail@yourdomain.tld
    volumes_from:
      - nginx-proxy
    volumes:
      - certs:/etc/nginx/certs:rw
      - acme:/etc/acme.sh
      - /var/run/docker.sock:/var/run/docker.sock:ro
    network_mode: bridge

volumes:
  conf:
  vhost:
  html:
  certs:
  acme:

rendered nginx configuration

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# configuration file /etc/nginx/nginx.conf:

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;

events {
    worker_connections  10240;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

# configuration file /etc/nginx/mime.types:

types {
    text/html                                        html htm shtml;
    text/css                                         css;
    text/xml                                         xml;
    image/gif                                        gif;
    image/jpeg                                       jpeg jpg;
    application/javascript                           js;
    application/atom+xml                             atom;
    application/rss+xml                              rss;

    text/mathml                                      mml;
    text/plain                                       txt;
    text/vnd.sun.j2me.app-descriptor                 jad;
    text/vnd.wap.wml                                 wml;
    text/x-component                                 htc;

    image/avif                                       avif;
    image/png                                        png;
    image/svg+xml                                    svg svgz;
    image/tiff                                       tif tiff;
    image/vnd.wap.wbmp                               wbmp;
    image/webp                                       webp;
    image/x-icon                                     ico;
    image/x-jng                                      jng;
    image/x-ms-bmp                                   bmp;

    font/woff                                        woff;
    font/woff2                                       woff2;

    application/java-archive                         jar war ear;
    application/json                                 json;
    application/mac-binhex40                         hqx;
    application/msword                               doc;
    application/pdf                                  pdf;
    application/postscript                           ps eps ai;
    application/rtf                                  rtf;
    application/vnd.apple.mpegurl                    m3u8;
    application/vnd.google-earth.kml+xml             kml;
    application/vnd.google-earth.kmz                 kmz;
    application/vnd.ms-excel                         xls;
    application/vnd.ms-fontobject                    eot;
    application/vnd.ms-powerpoint                    ppt;
    application/vnd.oasis.opendocument.graphics      odg;
    application/vnd.oasis.opendocument.presentation  odp;
    application/vnd.oasis.opendocument.spreadsheet   ods;
    application/vnd.oasis.opendocument.text          odt;
    application/vnd.openxmlformats-officedocument.presentationml.presentation
                                                     pptx;
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
                                                     xlsx;
    application/vnd.openxmlformats-officedocument.wordprocessingml.document
                                                     docx;
    application/vnd.wap.wmlc                         wmlc;
    application/wasm                                 wasm;
    application/x-7z-compressed                      7z;
    application/x-cocoa                              cco;
    application/x-java-archive-diff                  jardiff;
    application/x-java-jnlp-file                     jnlp;
    application/x-makeself                           run;
    application/x-perl                               pl pm;
    application/x-pilot                              prc pdb;
    application/x-rar-compressed                     rar;
    application/x-redhat-package-manager             rpm;
    application/x-sea                                sea;
    application/x-shockwave-flash                    swf;
    application/x-stuffit                            sit;
    application/x-tcl                                tcl tk;
    application/x-x509-ca-cert                       der pem crt;
    application/x-xpinstall                          xpi;
    application/xhtml+xml                            xhtml;
    application/xspf+xml                             xspf;
    application/zip                                  zip;

    application/octet-stream                         bin exe dll;
    application/octet-stream                         deb;
    application/octet-stream                         dmg;
    application/octet-stream                         iso img;
    application/octet-stream                         msi msp msm;

    audio/midi                                       mid midi kar;
    audio/mpeg                                       mp3;
    audio/ogg                                        ogg;
    audio/x-m4a                                      m4a;
    audio/x-realaudio                                ra;

    video/3gpp                                       3gpp 3gp;
    video/mp2t                                       ts;
    video/mp4                                        mp4;
    video/mpeg                                       mpeg mpg;
    video/quicktime                                  mov;
    video/webm                                       webm;
    video/x-flv                                      flv;
    video/x-m4v                                      m4v;
    video/x-mng                                      mng;
    video/x-ms-asf                                   asx asf;
    video/x-ms-wmv                                   wmv;
    video/x-msvideo                                  avi;
}

# configuration file /etc/nginx/conf.d/default.conf:
# nginx-proxy version : 1.4.0-57-ga3de83c
# Networks available to the container running docker-gen (which are assumed to
# match the networks available to the container running nginx):
#     bridge
# If we receive X-Forwarded-Proto, pass it through; otherwise, pass along the
# scheme used to connect to this server
map $http_x_forwarded_proto $proxy_x_forwarded_proto {
    default $http_x_forwarded_proto;
    '' $scheme;
}
map $http_x_forwarded_host $proxy_x_forwarded_host {
    default $http_x_forwarded_host;
    '' $host;
}
# If we receive X-Forwarded-Port, pass it through; otherwise, pass along the
# server port the client connected to
map $http_x_forwarded_port $proxy_x_forwarded_port {
    default $http_x_forwarded_port;
    '' $server_port;
}
# If the request from the downstream client has an "Upgrade:" header (set to any
# non-empty value), pass "Connection: upgrade" to the upstream (backend) server.
# Otherwise, the value for the "Connection" header depends on whether the user
# has enabled keepalive to the upstream server.
map $http_upgrade $proxy_connection {
    default upgrade;
    '' $proxy_connection_noupgrade;
}
map $upstream_keepalive $proxy_connection_noupgrade {
    # Preserve nginx's default behavior (send "Connection: close").
    default close;
    # Use an empty string to cancel nginx's default behavior.
    true '';
}
# Abuse the map directive (see <https://stackoverflow.com/q/14433309>) to ensure
# that $upstream_keepalive is always defined.  This is necessary because:
#   - The $proxy_connection variable is indirectly derived from
#     $upstream_keepalive, so $upstream_keepalive must be defined whenever
#     $proxy_connection is resolved.
#   - The $proxy_connection variable is used in a proxy_set_header directive in
#     the http block, so it is always fully resolved for every request -- even
#     those where proxy_pass is not used (e.g., unknown virtual host).
map "" $upstream_keepalive {
    # The value here should not matter because it should always be overridden in
    # a location block (see the "location" template) for all requests where the
    # value actually matters.
    default false;
}
# Apply fix for very long server names
server_names_hash_bucket_size 128;
# Default dhparam
ssl_dhparam /etc/nginx/dhparam/dhparam.pem;
# Set appropriate X-Forwarded-Ssl header based on $proxy_x_forwarded_proto
map $proxy_x_forwarded_proto $proxy_x_forwarded_ssl {
    default off;
    https on;
}
gzip_types text/plain text/css application/javascript application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
log_format vhost '$host $remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$upstream_addr"';
access_log off;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305';
    ssl_prefer_server_ciphers off;
error_log /dev/stderr;
resolver 192.168.10.1;
# HTTP 1.1 support
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $proxy_connection;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $proxy_x_forwarded_host;
proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
proxy_set_header X-Forwarded-Ssl $proxy_x_forwarded_ssl;
proxy_set_header X-Forwarded-Port $proxy_x_forwarded_port;
proxy_set_header X-Original-URI $request_uri;
# Mitigate httpoxy attack (see README for details)
proxy_set_header Proxy "";
server {
    server_name _; # This is just an invalid value which will never trigger on a real hostname.
    server_tokens off;
    access_log /var/log/nginx/access.log vhost;
    http2 on;
    listen 80;
    listen 443 ssl;
    ssl_session_cache shared:SSL:50m;
    ssl_session_tickets off;
    ssl_certificate /etc/nginx/certs/default.crt;
    ssl_certificate_key /etc/nginx/certs/default.key;
    return 503;
}

Containers logs

nginx-proxy-acme:

Info: running acme-companion version v2.2.9

nginx-proxy:

Info: running nginx-proxy version 1.4.0-57-ga3de83c
Setting up DH Parameters..
Warning: TRUST_DOWNSTREAM_PROXY is not set; defaulting to "true". For security, you should explicitly set TRUST_DOWNSTREAM_PROXY to "false" if there is not a trusted reverse proxy in front of this proxy.
Warning: The default value of TRUST_DOWNSTREAM_PROXY might change to "false" in a future version of nginx-proxy. If you require TRUST_DOWNSTREAM_PROXY to be enabled, explicitly set it to "true".
forego      | starting dockergen.1 on port 5000
forego      | starting nginx.1 on port 5100
nginx.1     | 2024/01/03 15:58:39 [notice] 21#21: using the "epoll" event method
nginx.1     | 2024/01/03 15:58:39 [notice] 21#21: nginx/1.25.3
nginx.1     | 2024/01/03 15:58:39 [notice] 21#21: built by gcc 12.2.0 (Debian 12.2.0-14) 
nginx.1     | 2024/01/03 15:58:39 [notice] 21#21: OS: Linux 3.10.105
nginx.1     | 2024/01/03 15:58:39 [notice] 21#21: getrlimit(RLIMIT_NOFILE): 524288:1048576
nginx.1     | 2024/01/03 15:58:39 [notice] 21#21: start worker processes
nginx.1     | 2024/01/03 15:58:39 [notice] 21#21: start worker process 26
nginx.1     | 2024/01/03 15:58:39 [notice] 21#21: start worker process 27
nginx.1     | 2024/01/03 15:58:39 [notice] 21#21: start worker process 28
nginx.1     | 2024/01/03 15:58:39 [notice] 21#21: start worker process 29
dockergen.1 | 2024/01/03 15:58:39 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification 'nginx -s reload'
dockergen.1 | 2024/01/03 15:58:39 Watching docker events
dockergen.1 | 2024/01/03 15:58:41 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification 'nginx -s reload'
dockergen.1 | 2024/01/03 15:58:41 Received event start for container cd9e6767f3a1
dockergen.1 | 2024/01/03 15:58:41 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification 'nginx -s reload'

Docker host

buchdag commented 8 months ago

Hi.

Does reverting to 2.2.8 fix the issue ?

buchdag commented 8 months ago

https://info.linuxserver.io/issues/2023-12-30-synology/

Known Issue: curl endless loop with 3.x Kernels

December 30, 2023 at 6:00 PM Known Issues synology alpine

◆ This issue is not resolved yet There is a known issue with the version of curl currently shipped with Alpine 3.19 and 3.x kernels, causing an endless loop to hang containers, usually on startup. The 3.x kernel line has been End of Life since 2017, but some prebuilt systems such as older Synology NAS units are still running it. At time of posting, around 10% of our images have already been rebased to Alpine 3.19 and are therefore affected by this issue, and this number will continue to rise over the next few months.

While a fix has been applied to the affected upstream library we do not have an ETA for if or when it will find its way into the version of curl shipped with Alpine 3.19.

Workaround As a workaround you can revert to an older tag from before the image was rebased to Alpine 3.19, but please be aware that older tags do not receive updates. Refer to the Changelog in the Readme to see when the change was made for a given image. For Unraid users this will require you to edit the container and under the Advanced options change the Repository to append the required tag.

Looks pretty much like the same issue to me 😞

vertIcod3r commented 8 months ago

Sure looks like it @buchdag and yes 2.2.8 is working fine. So have to wait for the fix to trickle down. Or see if updating to a newer kernel is possible. Thanks for finding this one.