Generated Let's Encrypt certificate invalid

[peterjaap]

We use this Docker container in our CI/CD process with Gitlab's Review Apps. We deploy it on a Kubernetes cluster and then request an LE certficate for the auto-generated hostname. But the certficate isn't valid;

Although my pipeline log says it's just fine;

[clientname-123-issue-slug.elgentos.io] > echo "clientname-123-issue-slug.elgentos.io" > /data/web/.dehydrated/domains.txt && dehydrated -c --create-dirs && hypernode-ssl-config-generator
[clientname-123-issue-slug.elgentos.io] < # INFO: Using main config file /etc/dehydrated/config
[clientname-123-issue-slug.elgentos.io] < + Generating account key...
[clientname-123-issue-slug.elgentos.io] < + Registering account key with ACME server...
[clientname-123-issue-slug.elgentos.io] < Processing clientname-123-issue-slug.elgentos.io
[clientname-123-issue-slug.elgentos.io] <  + Signing domains...
[clientname-123-issue-slug.elgentos.io] <  + Creating new directory /data/web/certs/clientname-123-issue-slug.elgentos.io ...
[clientname-123-issue-slug.elgentos.io] <  + Generating private key...
[clientname-123-issue-slug.elgentos.io] <  + Generating signing request...
[clientname-123-issue-slug.elgentos.io] <  + Requesting challenge for clientname-123-issue-slug.elgentos.io...
[clientname-123-issue-slug.elgentos.io] <  + Responding to challenge for clientname-123-issue-slug.elgentos.io...
[clientname-123-issue-slug.elgentos.io] <  + Challenge is valid!
[clientname-123-issue-slug.elgentos.io] <  + Requesting certificate...
[clientname-123-issue-slug.elgentos.io] <  + Checking certificate...
[clientname-123-issue-slug.elgentos.io] <  + Done!
[clientname-123-issue-slug.elgentos.io] <  + Creating fullchain.pem...
[clientname-123-issue-slug.elgentos.io] < INFO:kamikaze.sslconfig.main:Generating ssl config for clientname-123-issue-slug.elgentos.io
[clientname-123-issue-slug.elgentos.io] <  + Done!
[clientname-123-issue-slug.elgentos.io] < INFO:kamikaze.sslconfig.main:Generating ssl config for clientname-123-issue-slug.elgentos.io

[peterjaap]

Possibly related to #7, come to think of it.

[vdloo]

Cool that you're running this on kubernetes! As I mentioned in that other issue, it's probably the nginx config reloader that isn't being triggered. Usually inotify does that but that might not work in the Docker, so instead you can trigger it manually by running /usr/bin/nginx_config_reloader after making a change in /data/web/nginx

[peterjaap]

Yes, that was it. Thanks!

[peterjaap]

@vdloo ps I'll write a blog about our Hypernode Docker + Gitlab + k8s setup soon. It's awesome.

[peterjaap]

Extra step;

rm -rf /etc/nginx/ssl

Because otherwhise you'll run into this error when running /usr/bin/nginx_config_reloader;

2018/06/27 18:36:43 [emerg] 1504#1504: invalid server name or wildcard "www.*.hypernode.local" on 0.0.0.0:443

[peterjaap]

@vdloo unfortunately /usr/bin/nginx_config_reloader can only be run by root and we use the app user to deploy to (and thus request the LE certificate with).

[peterjaap]

The crontab shows;

* * * * * /usr/bin/nginx_config_reloader > /dev/null 2>&1 || true

So as a workaround, I thought about having the deployment wait a minute to have the cron catch up and generate the new nginx config after requesting the certificate. But it seems it doesn't run/work (it should create a magento.conf symlink to magento2.conf as found here);

root@hypernode-deployment-66b9d66b54-tdgbx /etc/nginx # date && ls -1lathr | grep magento
Wed Jun 27 19:45:05 UTC 2018
-rw-r--r-- 22 root root 3.1K Jun 22 14:21 magento1.conf
-rw-r--r-- 22 root root 3.6K Jun 22 14:21 magento2.conf

63 seconds later;

root@hypernode-deployment-66b9d66b54-tdgbx /etc/nginx # date && ls -1lathr | grep magento
Wed Jun 27 19:46:08 UTC 2018
-rw-r--r-- 22 root root 3.1K Jun 22 14:21 magento1.conf
-rw-r--r-- 22 root root 3.6K Jun 22 14:21 magento2.conf

When I run it manually, the magento.conf symlink does appear;

root@hypernode-deployment-66b9d66b54-tdgbx /etc/nginx # date && ls -1lathr | grep magento && /usr/bin/nginx_config_reloader && date && ls -1lathr | grep magento
Wed Jun 27 19:46:57 UTC 2018
-rw-r--r-- 22 root root 3.1K Jun 22 14:21 magento1.conf
-rw-r--r-- 22 root root 3.6K Jun 22 14:21 magento2.conf
2018-06-27 19:46:58,104 nginx_config_reloader INFO     /data/web/nginx
2018-06-27 19:46:58,276 nginx_config_reloader INFO     Reloading nginx config
Wed Jun 27 19:46:58 UTC 2018
-rw-r--r-- 22 root root 3.1K Jun 22 14:21 magento1.conf
-rw-r--r-- 22 root root 3.6K Jun 22 14:21 magento2.conf
lrwxrwxrwx  1 root root   24 Jun 27 19:46 magento.conf -> /etc/nginx/magento2.conf

[peterjaap]

Another workaround would be to have my deployment script, after the deployment, log back in as root and run /usr/bin/nginx_config_reloader but that's just dirty.

[JeroenBoersma]

So did you check if the cron daemon is running?

[vdloo]

I wouldn't be against putting passwordless sudo for the app user in the Docker image, that isn't real-hypernode-like but I can imagine that because there will be slight differences in the Docker anyway like this it would be practical. Then you could just run sudo /usr/bin/nginx_config_reloader as the app user

[peterjaap]

Yes that's also a possibility. We disabled password login anyway. But it seems the cron isn't running, although the daemon is there;

root@hypernode-deployment-66b9d66b54-tdgbx ~ # ps aux  |grep cron
root      1901  0.0  0.0   4240  1192 ?        Ss   Jun27   0:00 runsv cron
root      1934  0.0  0.0  35444  3044 ?        S    Jun27   0:00 /usr/sbin/cron -f

[vdloo]

I didn't mean password login for SSH, but 'passwordless sudo', which gives a user the right to run sudo for any command without being prompted for a password. I've proposed a PR in one of our private repos to add passwordless sudo for the app user in the vagrant and docker, but in the mean time you could already do this to achieve the same:

root@610fb4374ee0 /etc/sudoers.d # cat << EOF >> /etc/sudoers.d/app-nopasswd
> app ALL=(ALL) NOPASSWD:ALL
> EOF
root@610fb4374ee0 /etc/sudoers.d # su app -c 'sudo /usr/bin/nginx_config_reloader'
2018-06-29 08:35:19,299 nginx_config_reloader INFO     /data/web/nginx
2018-06-29 08:35:19,364 nginx_config_reloader INFO     Reloading nginx config

I'll let you know when this makes it into the image

[peterjaap]

Yes I already added it to our Dockerfile;

# Enable passwordless sudo for app user (see https://github.com/ByteInternet/hypernode-docker/issues/6)
RUN echo "app     ALL = (ALL) NOPASSWD: ALL" >> /etc/sudoers

This works. I'd vote against putting this in your image since not everybody would per se need it.

[vdloo]

Don't you think it would increase usability? Or would the down-side of making it less like a real Hypernode not be worth it?

By the way, the cron does work for me. Is there anything different about your setup that you can think of that might break this?

root@7f1642455f00 ~ # crontab -l | grep nginx_config
* * * * * /usr/bin/nginx_config_reloader > /dev/null 2>&1 || true
root@7f1642455f00 ~ # su app -c 'echo > /data/web/nginx/banaan'
root@7f1642455f00 ~ # ls /etc/nginx/app | grep banaan
banaan
root@7f1642455f00 ~ # ps aux | grep [c]ron
root       343  0.0  0.0   4240   604 ?        Ss   08:49   0:00 runsv cron
root       345  0.0  0.0  35444  2008 ?        S    08:49   0:00 /usr/sbin/cron -f

[peterjaap]

It would increase usability, but also adds a little bit more risk for environments that are semi-publicly available (like our staging environments).

Hm yes locally that works for me too, but not on the deployments we do on k8s. Weird.

[vdloo]

Alright I'll close that PR and leave it out for now, if someone else runs into this problem then I'll revisit