Use certificate for domain and subdomain

[janus-reith]

I have multiple apps running in one docker environment, and use nginx to route the main example.comto App 1, and sub.example.com to App 2. While this works so far, the letsencrypt certificate is just valid for sub.example.com.

Im using this configuration in my docker-compose file:

environment:
      - VIRTUAL_HOST=sub.example.com
      - LETSENCRYPT_HOST=sub.example.com
      - LETSENCRYPT_EMAIL=mail@example.com

and for App 2:

environment:
      - VIRTUAL_HOST=example.com
      - LETSENCRYPT_HOST=example.com
      - LETSENCRYPT_EMAIL=mail@example.com

How to use the proxy companion to properly generate certs for both hosts?

[janus-reith]

Ok, got it working now with the docker-compose configuraton described above:

I had to manually inspect the docker volumes, and delete the whole certs there. After restarting the containers, the certs are now generated properly.

But I think I'll still leave this open, as I don't consider this the proper solution. When I add new containers to my docker environment, the lets encrypt companion should realize that certs are missing, instead of just using the ones already generated.

[buchdag]

When I add new containers to my docker environment, the lets encrypt companion should realize that certs are missing

This is normally what it does. There was an issue in your case but it's impossible to find what caused it with just the environment from the two proxyed apps, so leaving this issue open without more information won't help much.

[ghost]

I am having an issue where I am creating containers using the same env as above, but just for docker run. The result is that it doesn't always (at random) create the right certs for the sub domains and therefore flags them as being invalid.

In my docker run command I have the correct domain I want, but it is not the domain being used for the certs

[buchdag]

Hi.

What version of the container are you using ? Can you provide the full docker run commands you are using to start your nginx-proxy stack and your proxyed containers, and if possible the log output of the letsencrypt container when it issues a certificate with the wrong domain set ?

[ghost]

I am using docker-compose version 1.18.0, build 8dd22a9 and Docker version 17.12.0-ce, build c97c6d6

I am a newbie at the log side of things - so not sure how to post those :-(

It was a late night, but when I used the docker-compose rather than docker run commands it works. If it didn't work I could just recreate the docker-letsencrypt-nginx-proxy-companion.

Before I started on the docker-compose route, I did this:

docker run -d \
  --name nginx-proxy \
  -p 80:80 -p 443:443 \
  -v ${PWD}/certs:/etc/nginx/certs:ro \
  -v /etc/nginx/vhost.d \
  -v /usr/share/nginx/html \
  -v /var/run/docker.sock:/tmp/docker.sock:ro \
  --label com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy \
  jwilder/nginx-proxy

docker run -d \
  --name nginx-letsencrypt \
  -v ${PWD}/certs:/etc/nginx/certs:rw \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  --volumes-from nginx-proxy \
  jrcs/letsencrypt-nginx-proxy-companion

docker run -d \
  --name mainweb \
  --expose=80 --expose=443 \
  -p 80 -p 443 \
  -e "VIRTUAL_HOST=sub.my-domain" \
  -e "LETSENCRYPT_HOST=sub.my-domain" \
  -e "LETSENCRYPT_EMAIL=my-email@example.com" \
  -v ${PWD}/mainweb:/usr/local/apache2/htdocs \
  httpd:2.2.34-alpine

This would work, but the problems started to come up when I started adding more containers, not always (and this was random) giving the right certificate.

The main container that was giving me issues where the combo of a mysql and phpmyadmin container. Where the phpmyadmin wouldn't get a certificate.

docker run -di \
  --name wp-cite-mysql \
  -p 3310:3306 \
  -e MYSQL_ROOT_PASSWORD="(password123)" \
  -e MYSQL_USER="admin" \
  -e MYSQL_PASSWORD="(Admin123)**" \
  -v ${PWD}/cite-mysql-2018-s1:/var/lib/mysql \
  mysql:5.7.20

docker run -d  \
  -p 80 -p 443 \
  --link wp-cite-mysql \
  -e "VIRTUAL_HOST=pma1.my-domain" \
  -e "LETSENCRYPT_HOST=pma1.my-domain" \
  -e "LETSENCRYPT_EMAIL=my-email" \
  phpmyadmin/phpmyadmin

[buchdag]

You shouldn't try to publish ports 80 or 443 ports on the proxyed containers with the -p argument, just expose the 80 port (or whatever port your container is serving http over) with the --expose argument if it's not already exposed by the Dockerfile (they are most probably already exposed). If you try to reach http serving containers with directly published ports, that won't go through the proxy and the certificate won't be served.

The --link is a legacy argument dating back from when docker networks didn't exist, I know it's unfortunately still on a lot of containers docs but its usage should be avoided in favor of said docker networks.

To get the logs of the container, do docker logs container-name. If you'd prefer no to publicly post your domains and subdomains (they'll appear in the logs) you can send me the logs by mail.

Do you use the exact same config with docker compose ?

[ghost]

Yeah - I do so I will look into it, the ports thing makes sense. I will also look into using networks, but as you said a lot of containers (and explanations on the web) still use them.

Sorry if this didn't answer the origin post.

[buchdag]

Sorry if this didn't answer the origin post.

No problem 😃

Also, If you have a working setup with docker-compose don't spend too much time trying to find why your first attempt with docker run did not work as intended. I'll give it a try with the same configuration as yours but I don't see any obvious reason for the proxy to randomly serve the wrong certificate.

[mvanalphen]

I seem to run into the exact same issue bcsjk11 is describing. I'm running a number of different sub-domains with their own certificates, using largely the same setup bcsjk11 is using (except for using a separate nginx-proxy) network. All sub-domains, such as jenkins.domain.com and sonarqube.domain.com, work properly.

However, the base domain (domain.com) appears to try to use the certificate of one of the sub-domains, I believe the most recently generated one.

For instance, these two URL's work properly: https://jenkins.carely.info, https://jira.carely.info However, https://carely.info/ is trying to use the certificate belonging to jenkins.carely.info.

docker-compose.yml of one of the subdomains:

version: '3'

services:
  jira:
    image: cptactionhank/atlassian-jira-software
    container_name: jira
    expose:
      - "8080"
    volumes:
      - /var/atlassian/jira:/var/atlassian/jira
    environment:
      - VIRTUAL_HOST=${HOSTNAME}.${DOMAINNAME}
      - LETSENCRYPT_HOST=${HOSTNAME}.${DOMAINNAME}
      - LETSENCRYPT_EMAIL=${LETSENCRYPT_EMAIL}
    restart: always

networks:
  default:
    external:
      name: nginx-proxy

docker-compose.yml of the main domain:

version: '3'

services:
  pts4-application:
    build: .
    container_name: pts4-application
    expose:
      - "80"
    environment:
      - VIRTUAL_HOST=${DOMAINNAME}
      - LETSENCRYPT_HOST=${DOMAINNAME}
      - LETSENCRYPT_EMAIL=${LETSENCRYPT_EMAIL}
    restart: always

networks:
  default:
    external:
      name: nginx-proxy

[buchdag]

Where does your compose file pick those variables from ? Are you sure they are set correctly ? Have you tried with hardcoded values ? Do you have any logs from the letsencrypt container to share ?

[mvanalphen]

The compose files pick the variables from a .env file in the same directory as the compose files. I've verified that the variables are set properly through docker inspect <container>. Unfortunately enough with hard-coded values it still doesn't seem to work.

I tried force renewing the certificates through docker exec nginx-letsencrypt /app/force_renew. The certificate for carely.info was generated properly, but then seems to use a different one instead.

The default.conf file is set as follows for the carely.info upstream, which seems to be incorrect:

# carely.info
upstream carely.info {
                ## Can be connect with "nginx-proxy" network
            # pts4-application
            server 172.23.0.2:80;
}
server {
    server_name carely.info;
    listen 80 ;
    access_log /var/log/nginx/access.log vhost;
    include /etc/nginx/vhost.d/default;
    location / {
        proxy_pass http://carely.info;
    }
}

I can't believe I hadn't checked the LetsEncrypt logs yet. The core issue seems to be in here:

2018-03-06 11:29:09,735:DEBUG:certbot.main:certbot version: 0.21.1
2018-03-06 11:29:09,737:DEBUG:certbot.main:Arguments: ['-q']
2018-03-06 11:29:09,738:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2018-03-06 11:29:09,763:DEBUG:certbot.log:Root logging level set at 30
2018-03-06 11:29:09,764:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2018-03-06 11:29:09,772:WARNING:certbot.renewal:expected /etc/letsencrypt/live/carely.info/cert.pem to be a symlink
2018-03-06 11:29:09,773:WARNING:certbot.renewal:Renewal configuration file /etc/letsencrypt/renewal/carely.info.conf is broken. Skipping.
2018-03-06 11:29:09,784:DEBUG:certbot.renewal:Traceback was:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/certbot/renewal.py", line 60, in _reconstitute
    renewal_candidate = storage.RenewableCert(full_path, config)
  File "/usr/lib/python3/dist-packages/certbot/storage.py", line 434, in __init__
    self._check_symlinks()
  File "/usr/lib/python3/dist-packages/certbot/storage.py", line 493, in _check_symlinks
    "expected {0} to be a symlink".format(link))
certbot.errors.CertStorageError: expected /etc/letsencrypt/live/carely.info/cert.pem to be a symlink

2018-03-06 11:29:09,784:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/certbot", line 11, in <module>
    load_entry_point('certbot==0.21.1', 'console_scripts', 'certbot')()
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1240, in main
    return config.func(config, plugins)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1142, in renew
    renewal.handle_renewal_request(config)
  File "/usr/lib/python3/dist-packages/certbot/renewal.py", line 443, in handle_renewal_request
    len(renew_failures), len(parse_failures)))
certbot.errors.Error: 0 renew failure(s), 1 parse failure(s)

Afterwards, I tried to fix the issue through this guide: https://github.com/certbot/certbot/issues/2550#issuecomment-197417732. Not sure if it's worked, as I've seem to hit the LetsEncrypt rate limit :P. Would that be the correct way of resolving this issue?

[buchdag]

Wait, what ? Why and how do you get those log output from certbot ?

We aren't using certbot at all in this container but simp_le.

[mvanalphen]

Where should I be retrieving the logs from in the actual container? I pulled these from /var/logs/letsencrypt from the host (which was strange anyway, because I don't have that folder mounted).

I did run a previous separate LetsEncrypt installation prior to using Docker which I uninstalled. Could that be causing issues? I don't recall ever using simp_le though.

[mvanalphen]

Okay, looks like python-certbot was still installed on the host. I've completely removed it, but the issue still persists after restarting the containers. The base domain still uses an incorrect certificate.

Where would one find the LetsEncrypt logs in the container?

[buchdag]

You can get them with docker logs name-or-id-of-the-container.

[mvanalphen]

Ah, there we go. Looks like the symlink was indeed the issue, which in turn was caused by my pre-Docker install of the Certbot. Once the ratelimit passes I think the URL will work fine.

My sincerest apologies for wasting your time like this!

[buchdag]

No apologies needed ! Could you detail how do you think your previous use of certbot led to a symlink issue ? That could be helpful to me on another issue (#337).

[mvanalphen]

For some reason, when installing nginx-proxy-companion, I decided to set the certs directory to the existing letsencrypt live directory (/etc/letsencrypt/live), as I had assumed I uninstalled certbot properly and it'd be easy to remember this directory on the host.

As it turns out, as I had previously already generated a certificate for the base domain, the carely.info folder already existed. This means that the symlinks could not be created by nginx-proxy-companion as it was pointing to an existing file.

As a result, the container failed to create a certificate for carely.info. And that leads to the issue you mentioned in #337: when the certificate generation fails, it uses the wrong certificate.

Disclaimer: This is still hypothetical, as I haven't been able to verify that removing certbot and the /etc/letsencrypt/live folder and having nginx-proxy-companion actually resolves the issue. This is because the certificate creation still fails due to rate limiting. But this seems like the most likely cause.

[buchdag]

No activity for 90+ days, closing.