Kiwix-serve should be able to be started with HTTPS

kelson42 commented 3 years ago

Because new Zimit ZIM files rely on Service workers which work only with HTTPS. Of course we would have to hardcode the certificate in the source-code.

This is necessary for example to get the HTTP daemon embedded within kiwix-desktop/android able to service ZIMit ZIM files.

kelson42 commented 3 years ago

@veloman-yunkan @mgautierfr Would that be easy to do?

mgautierfr commented 3 years ago

I'm not sure we should handle this (in fact, I don't want to).

We should not take care about the deployment of our software. We already have enough work to do. We will always need some kind of configuration from the user.

Adding https on top of a local http service is pretty easy with nginx[1], httpd or caddy[2]. Distributor should use them to correctly setup a production server.

We are not security people and I don't want to become one.

Of course we would have to hardcode the certificate in the source-code.

We really don't want to hardcode the certificate in the source-code. It would means that we distribute the certificate (and its private part). And, as certificate expires, we would have expiring binaries.

This is necessary for example to get the HTTP daemon embedded within kiwix-desktop/android able to service ZIMit ZIM files.

This is a big change in how we do things. It will turn our applications as "specialized browser" (aka webapp). kiwix-serve is far from being able to handle that. But anyway, even if we go this way, we don't need https, this is not needed on localhost for service worker.

[1] This is example configuration for nginx :

upstream backend_kiwix-serve{
    server 127.0.0.1:8080;
}

server {
    listen 443 ssl;
    ssl_certificate /path/to/certificate.pem 
    ssl_certificate_key /path/to/certificate_key.pem

    location / {
        include proxy_params;
        proxy_pass http://backend_kiwix-serve;
    }
}

[2] And the command line for caddy : sudo caddy reverse-proxy --from "you-domain.com" --to localhost:8080 It seems that you don't even need a certificate, caddy will get you one using let's encrypt. (And caddy is available all platforms, android included)

kelson42 commented 3 years ago

@mgautierfr We are not going to install nginx on the Kiwix users on Android... and same for Kiwix Desktop.

mgautierfr commented 3 years ago

What about caddy ? It is a "simple" binary written in go.

kelson42 commented 3 years ago

@mgautierfr Will have a look to caddy. Here is a piece of code to generate a self-signed certificate based on openSSL https://gist.github.com/nathan-osman/5041136

kelson42 commented 3 years ago

This is an attempt to clarify how this might be technically solved:

We need to listen on the HTTP (443) port in addition to the normal HTTP (80) port
To allow HTTPS we need to have a TLS compatible library bound to the HTTP server (should work more or less out of the box with libmicrotthpd)
To have HTTPS working we need to have a certificatem self-generated
This certificate should be delivered for IP address and/or a given local domain name
As consequence, the certificate should be created on the fly and can not be stored in the binary

@rgaudin @mgautierfr Does that sounds correct?

rgaudin commented 3 years ago

OK, I think we need to separate the certificate stuff from the HTTPS one. We could even release a first version with just HTTPS. I actually think it would help ensure we don't take too many shortcuts.

So we need an option to pass a certificate (path) to kiwix-serve. This would be used by anyone with a real certificate or want to use a trusted certificate. So that's great for local deployments. --certificate ?

In a second step, we'd want to help people without a certificate. Ideally, that would be a separate tool that's used to generate the certificate passed to kiwix-serve, but I'm not sure how practical that is.

That tool should be able to create a Certificate from a given Common Name but should also be capable of guessing IPs and hostnames of the machine and use that in SAN or Common Name.

Another use case would be for that tool to take a --root-cert path and sign generated certificates with that root CA. It will allow trsuting that root certificate to client devices yet use the generated mechanism. The use case I have in mind is BSF, creating a root CA for its deployments so their clients are not prompted with warnings. It's not top-noch security as they have to include their CA into their server and protect it, but it's way better than a hard-coded Kiwix CA or certificate in the repo.

The tool could even generate that CA cert (and client installers) in a different step so users don't have to get additional software for this.

kelson42 commented 3 years ago

@rgaudin I agree with the spliting. We should focus on just allowing HTTPS first and in this ticket.

JensKorte commented 3 years ago

Maybe it could be helpful to reuse some scripts from https://letsencrypt.org/getting-started/. Even if you create self-signed certificates.

With Shell Access

We recommend that most people with shell access use the Certbot ACME client. It can automate certificate issuance and installation with no downtime. It also has expert modes for people who don’t want autoconfiguration. It’s easy to use, works on many operating systems, and has great documentation. Visit the Certbot site to get customized instructions for your operating system and web server.

If Certbot does not meet your needs, or you’d like to try something else, there are many more ACME clients to choose from. Once you’ve chosen ACME client software, see the documentation for that client to proceed.

If you’re experimenting with different ACME clients, use our staging environment to avoid hitting rate limits.

kelson42 commented 3 years ago

@JensKorte This should be done in Kiwix Serve, without access to Internet and without Shell.

suterma commented 3 years ago

Disclaimer: I am not very involved yet in this project, but I would be with @mgautierfr 's comment from Nov 20, 2020:

Do not take responsibility on TLS security. It's a solved problem.

Configuratively doing HTTPS with a suitable server

For example, the pi-hole guys also did not do it for their admin web interface for a long time. And their solution is much more privacy-sensitive, imho. However, they have changed lately to use a configurative approach, see: https://discourse.pi-hole.net/t/enabling-https-for-your-pi-hole-web-interface/5771. If you (as project maintainer) want to encourage the use of https, you might go this path.

Using the sidecar pattern

As an alternative, which requires no changes on kiwix-serve, you could also promote the use of the sidecar pattern, which is basically a reverse proxy which exposes your endpoint via HTTS, taking responsibility for the TLS connection. Here's an article I used to actually do that for pi-hole in azure: https://docs.microsoft.com/en-us/azure/container-instances/container-instances-container-group-ssl

I would recommend going with the sidecar pattern.

ilos-vigil commented 2 years ago

Using Caddy isn't as straightforward as i expected, so i decide to leave Caddyfile which meant for local private/home network. I'm assuming you're running Caddy and kiwix-serve as non-root user.

{
  auto_https disable_redirects
}

https://:10001 {
  tls domain.crt domain.key {
    protocols tls1.2 tls1.2
  }
  reverse_proxy http://:8080
}

Some explanation:

auto_https disable_redirects, disable Caddy's automatic HTTPS redirect feature. By default Caddy will listen on port 80 for that feature. But since Caddy lack permission, it'll quit.
https://:10001, HTTPS port which will be accessed from another device.
http://:8080, kiwix-serve port.
tls domain.crt domain.key, load your own self-signed certificate with local IP address. In my case, i encounter SSL_ERROR_RX_RECORD_TOO_LONG with Caddy default certificate.
protocols tls1.2 tls1.2, specify TLS version in case your browser have problem with TLS 1.3.

kelson42 commented 2 years ago

@ilos-vigil We use Caddy on front for kiwix-serve in kiwix-hotspot. But, this is not going to fix anything here. It has to be in the binary itself.

ilos-vigil commented 2 years ago

@kelson42 Thanks for the information, but i don't use kiwix-hotspot and i don't mind setting Caddy manually.

mgautierfr commented 2 years ago

Making kiwix-serve handle ssl is feasible:

Adapt code https://www.gnu.org/software/libmicrohttpd/tutorial.html#Adding-a-layer-of-security (easy)
Make libkiwix depend on libgnutls and so compile libgnutls in our CI (which may be easy or not)

Managing the key and the certificate is left as a exercise to the user. Which lead to question for server started from kiwix-desktop or kiwix-android.

fketterman commented 1 year ago

So I am trying to use caddy with the following Caddyfile to allow reverse proxy to kiwix-serve:

192.168.1.164 { reverse_proxy 127.0.0.1:8080

tls /etc/caddy/192.168.1.164.pem /etc/caddy/192.168.1.164-key.pem }

No errors in caddy, and kiwix-serve does in fact come up from a different system on my network. However, all of the .zim files that require https give me a turning circle and do not error out nor display anything.

My question would be does the certificate need to have an actual CA signature? Or am I missing something else in this configuration?

rgaudin commented 1 year ago

If the certificate is trusted in the client browser, SW would work and thus the zimit ZIMs as well. You might want to turn on debug in Caddy and kiwix-serve and open your browser's debugging tools to check for any lead.

fketterman commented 1 year ago

rgaudin I ran this command: sudo caddy run --config /etc/caddy/Caddyfile to start caddy. I then browsed to my server and was able to access a zim file that uses the SW in the newer zim's. On the server I can see this error: 2023/06/18 14:54:32.808 ERROR http.handlers.reverse_proxy aborting with incomplete response {"error": "http2: stream closed"}

Note that ZIM's without the handlers work as expected, ones with get this error and a spinning circle on the page.

I also tried running the caddy command: sudo caddy reverse-proxy --from "192.168.1.164" --to localhost:8080

And received the same error message.

kelson42 commented 1 year ago

This ticket is not a forum to help user to configure their reverse proxy. It's not even the role of the Kiwix dev team to make support for Caddy, Apache or Ngix reverse-proxy mode.

evrial commented 1 year ago

HTTPS and service workers is super garbage idea, ZIM format is made for offline, is read only, why the heck should I stress with signing certificate and doing extra management with inevitable troubles for my local raspberry pi kiwix hosting? Please move to the drawing board and think about it.

kelson42 commented 1 year ago

I could agree, but nobody was able to propose a proper alternative to SW for Zimit!

Luckily a few months ago we have achieved to prototype zimit 2.0 without SW!

Therefore I will close this ticket.

evrial commented 1 year ago

God bless, sanity prevailed, love your work guys, wish that kiwix-serve had same slick UI as PWA.

go2tom42 commented 10 months ago

It seems that you don't even need a certificate, caddy will get you one using let's encrypt. (And caddy is available all platforms, android included)

Well for Caddy to get a certificate from let's encrypt Caddy would require internet access, and since Kiwix whole shtick is Internet Content Without Internet Access would that not be a major issue?

go2tom42 commented 10 months ago

This ticket is not a forum to help user to configure their reverse proxy. It's not even the role of the Kiwix dev team to make support for Caddy, Apache or Ngix reverse-proxy mode.

What is the goal of the Kiwix dev team?

I assume it's "Internet Content Without Internet Access" since that is the 1st thing you see when going to kiwix.org

HTTPS is required for some ZIM files HTTPS requires a SSL Certificate Using a proxy server provides HTTPS A member of the DEV team (mgautierfr) told users to use Caddy Using Caddy, Apache or Ngix all require internet access to get a SSL Certificate

see the problem?

kelson42 commented 10 months ago

It's not possible to get validate certificate without access to a CA which are only available on the Web.

This requirement is mostly here because our current ServiceWorker ZIM files based on Zimit requires to work over HTTP. Because Zimit 2.0 which will be ready in the next few months won't require HTTPS anymore, this feature request is not really pertinent anymore. Therefore this ticket is closed

kiwix / kiwix-tools

Kiwix-serve should be able to be started with HTTPS #416

Configuratively doing HTTPS with a suitable server

Using the sidecar pattern