search

ononoki1 / nginx-http3

Automatic latest NGINX mainline build with additional features for Debian bullseye.
https://blog.ononoki.org/enable-http3-for-nginx/
MIT License
44 stars 32 forks source link

OCSP Stapling doesn't work using Letsencrypt #28

Closed kamzata closed 2 years ago

kamzata commented 2 years ago

Using Letsencrypt I usually do:

ssl_stapling on;
ssl_stapling_verify on;

but now nginx -t says: 

nginx: [emerg] Read OCSP response file "/etc/nginx/" failed: -1 (SSL: error:02000015:system library:OPENSSL_internal:Is a directory error:11000002:BIO routines:OPENSSL_internal:system library)
nginx: configuration file /etc/nginx/nginx.conf test failed
ononoki1 commented 2 years ago

BoringSSL only supports using OCSP stapling via ssl_stapling_file. See here: https://github.com/kn007/patch#enable_boringssl_ocsppatch

However, OpenSSL does support these directives. You may use nginx-quictls if you need to use both HTTP/3 and direct OCSP stapling.

kamzata commented 2 years ago

nginx-quictls

I'll try for sure! Thanks! Does Nginx-quictls have the same modules enabled and removed as is for nginx-http3 package (except for BoringSSL of course)?

Forthermore, can I use the same nginx.conf configuration?

Using the configuration you posted (without enabling SSL_stapling) it works partially. I mean, it works now and then on Chrome and Firefox on MacOS. In Chrome and Firefox on Windows seem to work more often (usually after a page reload) while in Chrome and Firefox on iOS I receive ERR_INVALID_RESPONSE.

ononoki1 commented 2 years ago

Does Nginx-quictls have the same modules enabled and removed as is for nginx-http3 package (except for BoringSSL of course)?

Yes.

Forthermore, can I use the same nginx.conf configuration?

Quictls is based on OpenSSL, so your original config should work fine.

kamzata commented 2 years ago

How should I use the Nginx-quictls package? It's not a .deb

ononoki1 commented 2 years ago

Just use that to replace your nginx binary file. For Debian, its location is usually /usr/sbin/nginx.

kamzata commented 2 years ago

Just use that to replace your nginx binary file. For Debian, its location is usually /usr/sbin/nginx.

Thanks. Now the OCSP Stapling works. However, I'm encountering the same problems. It works (I'd say like a charm) on Chrome in Windows, but in Chrome and Firefox on iOS it doesn't. Just tried on Chrome for Android and it works too. Does it should work in iOS too?

By the way, should I add reuseport parameter?

ononoki1 commented 2 years ago

Does it should work in iOS too?

Yes. Maybe you can try other sites supporting HTTP/3 to see whether it's client's problem or server's.

By the way, should I add reuseport parameter?

Yes, it's recommended.

kamzata commented 2 years ago

You're right. I was proxy_pass http://xxx.xxx.xxx.xxx to Apache server. I tried another Apache virtual server and now it works on iOS too. That Apache server has 2 virtual servers with the same configuration:

<VirtualHost *:80>
        ServerAdmin admin@example.com
        DocumentRoot /var/www/html
        AllowEncodedSlashes NoDecode
        <Directory "/var/www/html">
                Options Indexes FollowSymLinks MultiViews
                AllowOverride All
                Require all granted
        </Directory>
        ErrorLog /var/log/apache2/test-error_log
</VirtualHost>

index.html (of /var/www/html which doesn't work)

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
  <!--
    Modified from the Debian original for Ubuntu
    Last updated: 2016-11-16
    See: https://launchpad.net/bugs/1288690
  -->
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
    <META NAME="robots" CONTENT="noindex,nofollow">
    <title>Apache2 Ubuntu Default Page: It works</title>
    <style type="text/css" media="screen">
  * {
    margin: 0px 0px 0px 0px;
    padding: 0px 0px 0px 0px;
  }

  body, html {
    padding: 3px 3px 3px 3px;

    background-color: #D8DBE2;

    font-family: Verdana, sans-serif;
    font-size: 11pt;
    text-align: center;
  }

  div.main_page {
    position: relative;
    display: table;

    width: 800px;

    margin-bottom: 3px;
    margin-left: auto;
    margin-right: auto;
    padding: 0px 0px 0px 0px;

    border-width: 2px;
    border-color: #212738;
    border-style: solid;

    background-color: #FFFFFF;

    text-align: center;
  }

  div.page_header {
    height: 99px;
    width: 100%;

    background-color: #F5F6F7;
  }

  div.page_header span {
    margin: 15px 0px 0px 50px;

    font-size: 180%;
    font-weight: bold;
  }

  div.page_header img {
    margin: 3px 0px 0px 40px;

    border: 0px 0px 0px;
  }

  div.table_of_contents {
    clear: left;

    min-width: 200px;

    margin: 3px 3px 3px 3px;

    background-color: #FFFFFF;

    text-align: left;
  }

  div.table_of_contents_item {
    clear: left;

    width: 100%;

    margin: 4px 0px 0px 0px;

    background-color: #FFFFFF;

    color: #000000;
    text-align: left;
  }

  div.table_of_contents_item a {
    margin: 6px 0px 0px 6px;
  }

  div.content_section {
    margin: 3px 3px 3px 3px;

    background-color: #FFFFFF;
    text-align: left;
  }

  div.content_section_text {
    padding: 4px 8px 4px 8px;

    color: #000000;
    font-size: 100%;
  }

  div.content_section_text pre {
    margin: 8px 0px 8px 0px;
    padding: 8px 8px 8px 8px;

    border-width: 1px;
    border-style: dotted;
    border-color: #000000;

    background-color: #F5F6F7;

    font-style: italic;
  }

  div.content_section_text p {
    margin-bottom: 6px;
  }

  div.content_section_text ul, div.content_section_text li {
    padding: 4px 8px 4px 16px;
  }

  div.section_header {
    padding: 3px 6px 3px 6px;

    background-color: #8E9CB2;

    color: #FFFFFF;
    font-weight: bold;
    font-size: 112%;
    text-align: center;
  }

  div.section_header_red {
    background-color: #CD214F;
  }

  div.section_header_grey {
    background-color: #9F9386;
  }

  .floating_element {
    position: relative;
    float: left;
  }

  div.table_of_contents_item a,
  div.content_section_text a {
    text-decoration: none;
    font-weight: bold;
  }

  div.table_of_contents_item a:link,
  div.table_of_contents_item a:visited,
  div.table_of_contents_item a:active {
    color: #000000;
  }

  div.table_of_contents_item a:hover {
    background-color: #000000;

    color: #FFFFFF;
  }

  div.content_section_text a:link,
  div.content_section_text a:visited,
   div.content_section_text a:active {
    background-color: #DCDFE6;

    color: #000000;
  }

  div.content_section_text a:hover {
    background-color: #000000;

    color: #DCDFE6;
  }

  div.validator {
  }
    </style>
  </head>
  <body>
    <div class="main_page">
      <div class="page_header floating_element">
        <img src="/icons/ubuntu-logo.png" alt="Ubuntu Logo" class="floating_element"/>
        <span class="floating_element">
          Apache2 Ubuntu Default Page
        </span>
      </div>
<!--      <div class="table_of_contents floating_element">
        <div class="section_header section_header_grey">
          TABLE OF CONTENTS
        </div>
        <div class="table_of_contents_item floating_element">
          <a href="#about">About</a>
        </div>
        <div class="table_of_contents_item floating_element">
          <a href="#changes">Changes</a>
        </div>
        <div class="table_of_contents_item floating_element">
          <a href="#scope">Scope</a>
        </div>
        <div class="table_of_contents_item floating_element">
          <a href="#files">Config files</a>
        </div>
      </div>
-->
      <div class="content_section floating_element">

        <div class="section_header section_header_red">
          <div id="about"></div>
          It works!
        </div>
        <div class="content_section_text">
          <p>
                This is the default welcome page used to test the correct
                operation of the Apache2 server after installation on Ubuntu systems.
                It is based on the equivalent page on Debian, from which the Ubuntu Apache
                packaging is derived.
                If you can read this page, it means that the Apache HTTP server installed at
                this site is working properly. You should <b>replace this file</b> (located at
                <tt>/var/www/html/index.html</tt>) before continuing to operate your HTTP server.
          </p>

          <p>
                If you are a normal user of this web site and don't know what this page is
                about, this probably means that the site is currently unavailable due to
                maintenance.
                If the problem persists, please contact the site's administrator.
          </p>

        </div>
        <div class="section_header">
          <div id="changes"></div>
                Configuration Overview
        </div>
        <div class="content_section_text">
          <p>
                Ubuntu's Apache2 default configuration is different from the
                upstream default configuration, and split into several files optimized for
                interaction with Ubuntu tools. The configuration system is
                <b>fully documented in
                /usr/share/doc/apache2/README.Debian.gz</b>. Refer to this for the full
                documentation. Documentation for the web server itself can be
                found by accessing the <a href="/manual">manual</a> if the <tt>apache2-doc</tt>
                package was installed on this server.

          </p>
          <p>
                The configuration layout for an Apache2 web server installation on Ubuntu systems is as follows:
          </p>
          <pre>
/etc/apache2/
|-- apache2.conf
|       `--  ports.conf
|-- mods-enabled
|       |-- *.load
|       `-- *.conf
|-- conf-enabled
|       `-- *.conf
|-- sites-enabled
|       `-- *.conf
          </pre>
          <ul>
                        <li>
                           <tt>apache2.conf</tt> is the main configuration
                           file. It puts the pieces together by including all remaining configuration
                           files when starting up the web server.
                        </li>

                        <li>
                           <tt>ports.conf</tt> is always included from the
                           main configuration file. It is used to determine the listening ports for
                           incoming connections, and this file can be customized anytime.
                        </li>

                        <li>
                           Configuration files in the <tt>mods-enabled/</tt>,
                           <tt>conf-enabled/</tt> and <tt>sites-enabled/</tt> directories contain
                           particular configuration snippets which manage modules, global configuration
                           fragments, or virtual host configurations, respectively.
                        </li>

                        <li>
                           They are activated by symlinking available
                           configuration files from their respective
                           *-available/ counterparts. These should be managed
                           by using our helpers
                           <tt>
                                a2enmod,
                                a2dismod,
                           </tt>
                           <tt>
                                a2ensite,
                                a2dissite,
                            </tt>
                                and
                           <tt>
                                a2enconf,
                                a2disconf
                           </tt>. See their respective man pages for detailed information.
                        </li>

                        <li>
                           The binary is called apache2. Due to the use of
                           environment variables, in the default configuration, apache2 needs to be
                           started/stopped with <tt>/etc/init.d/apache2</tt> or <tt>apache2ctl</tt>.
                           <b>Calling <tt>/usr/bin/apache2</tt> directly will not work</b> with the
                           default configuration.
                        </li>
          </ul>
        </div>

        <div class="section_header">
            <div id="docroot"></div>
                Document Roots
        </div>

        <div class="content_section_text">
            <p>
                By default, Ubuntu does not allow access through the web browser to
                <em>any</em> file apart of those located in <tt>/var/www</tt>,
                <a href="http://httpd.apache.org/docs/2.4/mod/mod_userdir.html" rel="nofollow">public_html</a>
                directories (when enabled) and <tt>/usr/share</tt> (for web
                applications). If your site is using a web document root
                located elsewhere (such as in <tt>/srv</tt>) you may need to whitelist your
                document root directory in <tt>/etc/apache2/apache2.conf</tt>.
            </p>
            <p>
                The default Ubuntu document root is <tt>/var/www/html</tt>. You
                can make your own virtual hosts under /var/www. This is different
                to previous releases which provides better security out of the box.
            </p>
        </div>

        <div class="section_header">
          <div id="bugs"></div>
                Reporting Problems
        </div>
        <div class="content_section_text">
          <p>
                Please use the <tt>ubuntu-bug</tt> tool to report bugs in the
                Apache2 package with Ubuntu. However, check <a
                href="https://bugs.launchpad.net/ubuntu/+source/apache2"
                rel="nofollow">existing bug reports</a> before reporting a new bug.
          </p>
          <p>
                Please report bugs specific to modules (such as PHP and others)
                to respective packages, not to the web server itself.
          </p>
        </div>

      </div>
    </div>
    <div class="validator">
    </div>
  </body>
</html>

The only change is the DocumentRoot directive which point to another folder. So, why does one it's working and the other not?

kamzata commented 2 years ago

I spoke too soon... it crashes even using the other website. It seems really unstable. Furthermore, using auth_basic it switches often from HTTP/2 to HTTP/3 like a mad. Is it really unstable as it seems or I'm wrong?

ononoki1 commented 2 years ago

Is it really unstable as it seems or I'm wrong?

I never use Apple product, so frankly speaking I don't know. XD

kamzata commented 2 years ago

Is it really unstable as it seems or I'm wrong?

I never use Apple product, so frankly speaking I don't know. XD

That's ok but do you find it enough stable in the systems you use? It crashed often even in Chrome or Firefox on Windows and on Android. Is it just my experience? Do you use a reverse proxy configuration?

ononoki1 commented 2 years ago

It crashed often even in Chrome or Firefox on Windows and on Android.

What do you mean by crash? Does nginx exit unexpectedly?

I never meet such cases myself. IMO it's stable enough.

Do you use a reverse proxy configuration?

Yes. Here is one of my dynamic sites using proxy_pass: https://search.ononoki.org. Maybe you can test whether you use HTTP/3 normally on this site via various clients.

kamzata commented 2 years ago

What do you mean by crash? Does nginx exit unexpectedly?

E.g. after I browse some pages (maybe after 20/40 seconds) Chrome says ERR_QUIC_PROTOCOL_ERROR, then the page reload by itself and some resources are loaded using HTTP/2, some others using HTTP/3, and some other fail to load.

I never meet such cases myself. IMO it's stable enough.

I'm glad to hear that. Likely it's my configuration wrong.

Yes. Here is one of my dynamic sites using proxy_pass: https://search.ononoki.org. Maybe you can test whether you use HTTP/3 normally on this site via various clients.

I tried and I can reach it in HTTP/3 and browse it without "crash".

/var/log/nginx/error.log

2022/07/15 02:24:42 [error] 3039#3039: *557 failed to send stream cancellation while waiting for request, client: 80.117.XXX.XXX, server: 0.0.0.0:443
2022/07/15 02:25:26 [error] 3039#3039: *803 failed to send stream cancellation while closing request, client: 80.117.XXX.XXX, server: 0.0.0.0:443
ononoki1 commented 2 years ago

So, it seems that the problem happens on the server side. You can show the corresponding nginx config (delete sensitive information first). Maybe I can do something for you.

kamzata commented 2 years ago

So, it seems that the problem happens on the server side. You can show the corresponding nginx config (delete sensitive information first). Maybe I can do something for you.

nginx.conf

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;

events {
    use epoll;
    worker_connections  1024;
    multi_accept on;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    ##
    # Logging Settings
    ##

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    ##
    # Basic Settings
    ##

aio threads;
  aio_write on;
  brotli on;
  brotli_types application/atom+xml application/javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-opentype application/x-font-truetype application/x-font-ttf application/x-javascr>
  etag off;
  gzip on;
  gzip_comp_level 6;
  gzip_types application/atom+xml application/javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-opentype application/x-font-truetype application/x-font-ttf application/x-javascrip>
  more_clear_headers server;
  quic_gso on;
  quic_retry on;
  sendfile on;
  tcp_nopush on;
  include /etc/nginx/conf.d/*.conf;

}

conf.d/dev.example.com.conf

server {
    if ($host = dev.example.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    listen   80;
#    listen   [::]:80;
    server_name dev.example.com;
}

server {
    listen          443 ssl http2;
#    listen          [::]:443 ssl http2;
    listen          443 http3;
#    listen          [::]:443 http3;

    server_name     dev.example.com;

    add_header Alt-Svc 'h3=":443"; ma=3600';

    auth_basic              "Protected Area";
    auth_basic_user_file    .htpasswd;

    include ssl-params.conf;

    location / {
        proxy_pass  http://192.168.XXX.XXX;
        include proxy_params.conf;

        proxy_buffers              128 8k;
        proxy_buffer_size          512k;
        proxy_busy_buffers_size    512k;

        add_header Alt-Svc 'h3=":443"; ma=3600';

    }

#   include snippets/self-signed.conf;

    ssl_certificate /etc/letsencrypt/live/dev.example.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/dev.example.com/privkey.pem; # managed by Certbot
}

ssl-params.conf

ssl_session_timeout 1d;
ssl_session_cache shared:SSL:10m;
ssl_buffer_size 1400;
ssl_session_tickets off;

ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305;
ssl_prefer_server_ciphers off;
ssl_ecdh_curve secp384r1;
ssl_dhparam /etc/nginx/dhparam.pem;

ssl_stapling on;
ssl_stapling_verify on;

resolver 1.1.1.1 1.0.0.1 valid=300s;
resolver_timeout 5s;

ssl_early_data on;

proxy_request_buffering off;
quic_retry on;
ononoki1 commented 2 years ago

Try these things.

  1. Add reuseport parameter to listen directive, e.g. listen 443 http3 reuseport;.
  2. Add always to Alt-Svc header, e.g. add_header Alt-Svc 'h3=":443"; ma=3600' always;
kamzata commented 2 years ago

Try these things.

  1. Add reuseport parameter to listen directive, e.g. listen 443 http3 reuseport;.
  2. Add always to Alt-Svc header, e.g. add_header Alt-Svc 'h3=":443"; ma=3600' always;

Similar result. Now the page is loaded using HTTP/2 while the resources are loaded on HTTP/3. Sometime, I think when it tries to using the HTTP/3 to get the page, Chrome says ERR_TOO_MANY_REDIRECTS. I don't see anymore the ERR_QUIC_PROTOCOL_ERROR.

ononoki1 commented 2 years ago

ERR_TOO_MANY_REDIRECTS is strange if it's not caused by backend service but HTTP/3. Have you tried static site? I mean without proxy_pass.

kamzata commented 2 years ago

ERR_TOO_MANY_REDIRECTS is strange if it's not caused by backend service but HTTP/3. Have you tried static site? I mean without proxy_pass.

No, I haven't. However the response header with status code 302 has the location pointing always at https://dev.example.com/. Ok, I'll try to point the location directive to a local folder without using proxy_pass.

kamzata commented 2 years ago

I tried to run this simple website locally (using root directive) and it loads through HTTP/3 without any issue. I'll try to move it on the backend server to check. What do you think?

ononoki1 commented 2 years ago

I'm not sure whether it's your backend service's problem or nginx's proxy parameter's. Can you please also show the proxy_params.conf file?

kamzata commented 2 years ago

I'm not sure whether it's your backend service's problem or nginx's proxy parameter's. Can you please also show the proxy_params.conf file?

I moved the test website to the backend server. Here's the behavior:

I don't have any Linux desktop distro right now to try but I think it should be work too. If I run it locally using root directive it works everywhere. However, your site which use proxy_pass it works everywhere too.

proxy_params.conf

proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Accept-Encoding "";

add_header              Front-End-Https   on;
proxy_redirect off;
ononoki1 commented 2 years ago

Try changing proxy_set_header Host $http_host; to proxy_set_header Host $host;.

kamzata commented 2 years ago

Try changing proxy_set_header Host $http_host; to proxy_set_header Host $host;.

Nothing's changed. How does your proxy params look like?

ononoki1 commented 2 years ago

How does your proxy params look like?

server {
  listen 443 ssl http2 reuseport;
  listen 443 http3 reuseport;
  server_name search.ononoki.org;
  location / {
    proxy_http_version 1.1;
    proxy_pass http://ip:port;
    proxy_set_header Connection '';
    proxy_set_header Host $host;
  }
}
kamzata commented 2 years ago

How does your proxy params look like?

server {
  listen 443 ssl http2 reuseport;
  listen 443 http3 reuseport;
  server_name search.ononoki.org;
  location / {
    proxy_http_version 1.1;
    proxy_pass http://ip:port;
    proxy_set_header Connection '';
    proxy_set_header Host $host;
  }
}

Comment all directives except proxy_set_header Host $host; from proxy_params.conf seems to work. But whenever I add any proxy_set_header, add_header or proxy_redirect off seems not work anymore.

Edit: it worked for a while, now it doesn’t work anymore

Edit2: I don’t think any cache is involved but I could be wrong of course.

kamzata commented 2 years ago

Do you have HTTP/2 enabled on the backend server? Are you using Apache there?

kamzata commented 2 years ago

Done!

This is what I added/changed:

However, I've got another question for you @ononoki1: testing your website with this tool I can see the handshake is done in ~10 ms (and packet RX ~3 ms). Testing my website I can see is done in ~270 ms (and packet RX ~86 ms). How can I improve those values?

ononoki1 commented 2 years ago

Congratulations!

How can I improve those values?

Try enabling ssl_session_tickets.

kamzata commented 2 years ago

Try enabling ssl_session_tickets.

Done but it didn't help so much (~265ms). Are you using nginx-http3 or nginx-quictls with Letsencrypt on search.ononoki.org? If you are using nginx-http3, did you enabled OCSP_Stapling? How? Could be also this the reason of the performance difference between my server and yours?

ononoki1 commented 2 years ago

Are you using nginx-http3 or nginx-quictls with Letsencrypt on search.ononoki.org?

I'm using nginx-quictls.

Could be also this the reason of the performance difference between my server and yours?

I tend to think that the reason is geographic distance.

kamzata commented 2 years ago

Hello @ononoki1, does your build have these support too?

I'm comparing it with this package.

ononoki1 commented 2 years ago

No because of performance issue (server response slows down a bit according to my test).

If you want to enable these support, you can simply use this patch and this one.

kamzata commented 2 years ago

No because of performance issue (server response slows down a bit according to my test).

If you want to enable these support, you can simply use this patch and this one.

Thanks. How can I apply those patch in case?

One more thing: why did you choose ssl_ecdh_curve X25519:P-256:P-384? I was using secp384r1. What's the difference?

ononoki1 commented 2 years ago

How can I apply those patch in case?

Similar to here.

why did you choose ssl_ecdh_curve X25519:P-256:P-384?

P-256 and P-384 are NIST curves, which may contain backdoors from NSA. See here. So, X25519 is better.

kamzata commented 2 years ago

How I can set reuseport on different servers?

ononoki1 commented 2 years ago

reuseport applies to ports, so if your servers share the same port (e.g. 443), you can and only need to set reuseport to one of them.

kamzata commented 2 years ago

reuseport applies to ports, so if your servers share the same port (e.g. 443), you can and only need to set reuseport to one of them.

However, I still have problems for HTTP/3 on some Wordpress websites which running Apache on the backend. Even though I set on nginx proxy_hide_header Upgrade and Header unset Upgrade on Apache backend the HTTP/3 connection fails and the nginx log says:

2022/07/19 14:29:45 [error] 7982#7982: *31052 failed to send stream cancellation while waiting for request, client: 80.117.XXX.XXX, server: 0.0.0.0:443

This happen just in HTTP/3. Any hints?

This is the header returned from Apache backend server from nginx:

nginx  ⌁ root  /etc/nginx  130  curl -I http://192.168.1.111
HTTP/1.1 301 Moved Permanently
Date: Tue, 19 Jul 2022 12:39:18 GMT
Server: Apache/2.4.54 (Debian)
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
X-Redirect-By: WordPress
Set-Cookie: PHPSESSID=4i59ia4ubadv74doeodov74858; path=/
Location: https://www.example.com/
Referrer-Policy: no-referrer-when-downgrade
Content-Type: text/html; charset=UTF-8
kamzata commented 2 years ago

If I set reuseport on the default server block http3 doesn't work anymore on any server. Furthermore, if I remove reuseport just the first website works in HTTP/3, the other one throws errors like those:

2022/07/19 18:17:47 [alert] 14500#14500: *51165 open socket #10 left in connection 81
2022/07/19 18:17:47 [alert] 14500#14500: aborting
2022/07/19 18:17:49 [error] 14853#14853: *51483 SSL_do_handshake() failed (SSL: error:0A000102:SSL routines::unsupported protocol) while handling frames, client: 80.117.XXX.XXX, server: 0.0.0.0:443

By the way, I replaced the Wordpress installation with the test website I used before and I installed the same modules on both backend Apache servers. They differ just for the Apache version: example1.com (Apache 2.4.54 on Debian), example2.com (Apache 2.4.41 on Ubuntu)

Here's the config:

  server {
    listen 80 reuseport;
    listen [::]:80 reuseport;
    return 444;
  }
  server {
    listen 443 reuseport ssl http2;
    listen [::]:443 reuseport ssl http2;
    listen 443 reuseport http3;
    listen [::]:443 reuseport http3;
    ssl_reject_handshake on;
  }

server {
    if ($host = example1.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    if ($host = www.example1.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

        listen   80;
        listen   [::]:80;
        server_name  www.example1.com example1.com;
}

server {
    if ($host = example1.com) {
        return 301 https://www.$host$request_uri;
    }

    listen          443 ssl http2;
    listen          [::]:443 ssl http2;
    listen          443 http3;
    listen          [::]:443 http3;

    server_name     www.example1.com example1.com;

    include ssl-params.conf;

    location / {
        proxy_pass  http://192.168.1.111/;
        include proxy_params.conf;

        add_header Alt-Svc 'h3=":443"; ma=3600' always;
    }

    ssl_certificate /etc/letsencrypt/live/example1.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/example1.com/privkey.pem; # managed by Certbot
}

server {
    if ($host = example2.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    if ($host = www.example2.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

        listen   80;
        listen   [::]:80;
        server_name  www.example2.com example2.com;
}

server {
    if ($host = example2.com) {
        return 301 https://www.$host$request_uri;
    }

    listen          443 ssl http2;
    listen          [::]:443 ssl http2;
    listen          443 http3;
    listen          [::]:443 http3;

    server_name     www.example2.com example2.com;

    include ssl-params.conf;

    location / {
        proxy_pass  http://192.168.1.111/;
        include proxy_params.conf;

        add_header Alt-Svc 'h3=":443"; ma=3600' always;
    }

    ssl_certificate /etc/letsencrypt/live/example2.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/example2.com/privkey.pem; # managed by Certbot
}

Do you use multiple server blocks?

ononoki1 commented 2 years ago

I'm not familiar with Apache httpd or WordPress, but I do have dynamic sites with PHP (though not WordPress). I use nginx and php-fpm for these sites without issues. An example is https://paste.ononoki.org.

Do you use multiple server blocks?

Yes.

kamzata commented 2 years ago

I'm not familiar with Apache httpd or WordPress, but I do have dynamic sites with PHP (though not WordPress). I use nginx and php-fpm for these sites without issues. An example is https://paste.ononoki.org.

Do you use multiple server blocks?

Yes.

Do you use a NAT configuration? Are you on a LXC Container?

I moved the working container with the test website behind this reverse proxy too and now it doesn't work anymore. It's like the first server takes all traffic from 443/UDP port.

ononoki1 commented 2 years ago

Do you use a NAT configuration? Are you on a LXC Container?

No NAT. My server is KVM virtualized.

kamzata commented 2 years ago

If I use reuseport on the default server no one of the websites is reached in http/3. Other than add the header add_header Alt-Svc 'h3=":443"; ma=3600' always; what can I do? I can see it in the response but Chrome or Firefox won't use HTTP/3. I tried to put it at the server block level as well but it's like ignored there.

I also do some netcat tests to check if the UDP packets are received and it seems all ok. Any ideas?

Furthermore, trying to check if HTTP/3 support is enabled with this site, the response is negative (QUIC connection could not be established) and in the nginx error.log file I can see:

2022/07/22 02:42:00 [error] 7477#7477: *19222 SSL_do_handshake() failed (SSL: error:0A000102:SSL routines::unsupported protocol) while handling frames, client: 2604:a880:800:a1::1279:3001, server: [::]:443
kamzata commented 2 years ago

Solved! The problem was the SSL directives placement. I was using them (like this include ssl-params.conf;) at the server block level. Moving them to the http block level solved all issues! I don't know why HTTP/2 worked well anyway! Hope don't arise other problems! Thanks for your support.

kamzata commented 2 years ago

One more thing: do you update your nginx-quic manually? O do you have a bash script which checks, download, and replace the latest nginx-quic binary?

ononoki1 commented 2 years ago

do you update your nginx-quic manually?

When there is a new release, I first check where the changes come from (nginx-quic or dependency modules) and their precise content, then I think about whether it's necessary to upgrade. If yes, I will upgrade it. So obviously such process cannot be automated. And I am strongly against upgrading without knowing the changes first.