Certs in Secret are generated every minute

[Vad1mo]

I have setup docker-flow-proxy-letsencrypt to use docker secrets, everything is working except that the certs are generated on each redeploy or each minute(if there are requests).

Example:

Secret Name | Creation Date
whoami.exmaple.com.pem-20171225-235020 | 2017-12-26 00:50:20 
whoami.exmaple.com.pem-20171225-234920 | 2017-12-26 00:49:20 
whoami.exmaple.com.pem-20171225-234820 | 2017-12-26 00:48:20 

Alle secrets have different Private and Public Cert.

I created the Proxy relatively straight forward from the examples. Only difference is the network name proxy_net and the constraints on manager.

proxy-stack.yml

version: "3"
services:
  proxy:
    image: vfarcic/docker-flow-proxy
    ports:
      - 80:80
      - 443:443
    networks:
      - net
    environment:
      - LISTENER_ADDRESS=swarm-listener
      - MODE=swarm
      - SERVICE_NAME=proxy_proxy
    deploy:
      mode: global
      placement:
        constraints: [node.role == worker]

  swarm-listener:
    image: vfarcic/docker-flow-swarm-listener
    networks:
      - net
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - DF_NOTIFY_CREATE_SERVICE_URL=http://proxy-le:8080/v1/docker-flow-proxy-letsencrypt/reconfigure
      - DF_NOTIFY_REMOVE_SERVICE_URL=http://proxy_proxy:8080/v1/docker-flow-proxy/remove
    deploy:
      placement:
        constraints: [node.role == manager]

  proxy-le:
    image: nib0r/docker-flow-proxy-letsencrypt
    networks:
      - net
    environment:
      - DF_PROXY_SERVICE_NAME=proxy_proxy
      - CERTBOT_OPTIONS=--staging
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    deploy:
      replicas: 1
      placement:
        constraints: [node.role == manager]      
      labels:
        - com.df.notify=true
        - com.df.distribute=true
        - com.df.servicePath=/.well-known/acme-challenge
        - com.df.port=8080
networks:
  net:

The sample application I am trying to deploy. The only difference here is that I force the redirect from http to https.

version: "3"
services:
  whoami:
    image: jwilder/whoami
    networks:
      - proxy_net
    deploy:
      replicas: 1
      labels:
        - com.df.notify=true
        - com.df.distribute=true
        - com.df.serviceDomain=whoami.exmaple.com
        #- com.df.servicePath=/
        #- com.df.srcPort=443
        - com.df.port=8000
        - com.df.httpsOnly=true
        - com.df.httpsRedirectCode=301        
        - com.df.letsencrypt.host=whoami.exmaple.com
        - com.df.letsencrypt.email=vb@8gears.com
networks:
  proxy_net:
    external: true

In the docs I found this Info Block about Certs

Info Since many other types of information can be stored as secrets, Docker Flow Proxy assumes that secrets that should be used as certificates are prefixed with cert- or cert_. Secrets with any other naming convention will not be loaded as certificates.

Maybe this is the reason the flow proxy cant find the certs and recreates.

[n1b0r]

Hello,

Isn't your service being rescheduled every minute which is triggering a new certificate ? Could you provide logs of dfple ?

When a certificate is generated or updated, a new secret is created using a name like domain-%Y%m%d-%H%M%S with a limitation of 48 characters for the domain. When a secret is attached to a service, the mounted file is named cert-* as described in dfp doc.

[Vad1mo]

ok, I understand, yes you are right the service is rescheduled quite often sometimes. What I still don't understand why does the secret needs to be recreated? It is quite confusing to clean up the secrets afterwards. While rolling out 3 services with trial and errors I got about 100 certs created. And now that services are rescheduled every now and then a new secret is created.

Isn't there an option where secrets aren't recreated all the times but rather reused?

[n1b0r]

As written in the secret documentation you must have a volume binded to /etc/letsencrypt in order to keep certificates persitent, and that docker-flow-proxy-letsencrypt will not regen a new one each times it (dfple) is recreated. Then we create a new secret each time a cert is generated/updated and attach it to the docker-flow-proxy service. We have to create a new secret each time the cert change because docker secrets are immutable, we cannot update them once created.

Hope this is clear, do not hesitate to comment if you have any troubles.

[Vad1mo]

ahh ok, I was reading the documentation but for me the secrets represented the concept of a persistent storage (for secrets). So I was reading it as mandatory unless you are using secrets.

Thanks for pointing it out again. If secrets are enabled, then their purpose is to provide the same certs for other services to consume and not for themself.

[jaschaio]

I am using secrets and a persistent volume as explained in http://proxy-letsencrypt.dockerflow.com/example-secrets/. But still my certificates are regenerated each time dfple is restarted. Seems like a bug to me.

[Vad1mo]

@jaschaio is you volume correctly mounted by dfple on start?

[jaschaio]

@Vad1mo Yes it's correctly mounted to /etc/letsencrypt.

And if I check the contents of the volume I can see that the certificates are correctly added to the volume.

This only seems to happen if I completely remove the stack and start it again. If I update one of the services it doesn't happen.

Logs after restarting:

2018-04-16T07:48:51.902443690Z proxy_letsencrypt.1.nqqis0uabm49@Docker-Manager    | crond: crond (busybox 1.24.2) started, log level 8
2018-04-16T07:48:58.332260564Z proxy_letsencrypt.1.nqqis0uabm49@Docker-Manager    | 2018-04-16 07:48:58,330;INFO;request for service: whoami
2018-04-16T07:48:58.357287484Z proxy_letsencrypt.1.nqqis0uabm49@Docker-Manager    | 2018-04-16 07:48:58,355;INFO;request for service: proxy_letsencrypt
2018-04-16T07:48:58.493297410Z proxy_letsencrypt.1.nqqis0uabm49@Docker-Manager    | 2018-04-16 07:48:58,491;INFO;request for service: registry_registry
2018-04-16T07:48:58.493340177Z proxy_letsencrypt.1.nqqis0uabm49@Docker-Manager    | 2018-04-16 07:48:58,491;INFO;letsencrypt support enabled.
2018-04-16T07:48:58.493344918Z proxy_letsencrypt.1.nqqis0uabm49@Docker-Manager    | 2018-04-16 07:48:58,492;INFO;Letsencrypt support enabled, processing request: domains=registry.example.com email=jascha.brinkmann@gmail.com testing=None
2018-04-16T07:49:00.139715302Z proxy_letsencrypt.1.nqqis0uabm49@Docker-Manager    | 2018-04-16 07:49:00,139;INFO;creating secret for cert registry.example.com.pem
2018-04-16T07:49:00.156781817Z proxy_letsencrypt.1.nqqis0uabm49@Docker-Manager    | 2018-04-16 07:49:00,156;INFO;attaching secret registry.example.com.pem-20180416-074900
2018-04-16T07:49:00.166897622Z proxy_letsencrypt.1.nqqis0uabm49@Docker-Manager    |   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2018-04-16T07:49:00.167584280Z proxy_letsencrypt.1.nqqis0uabm49@Docker-Manager    |                                  Dload  Upload   Total   Spent    Left  Speed
2018-04-16T07:49:00.181049851Z proxy_letsencrypt.1.nqqis0uabm49@Docker-Manager    | {"Warnings":null}
100  1411  100    18  100  1393     18   1393  0:00:01 --:--:--  0:00:01  125k    | 

Output of docker secret ls

m0kdef57vfexqpv6wv4qwexrq   registry.example.com.pem-20180416-072247                       28 minutes ago      28 minutes ago
f7sgy8k2bkes4my0rbizstgxv   registry.example.com.pem-20180416-074900                       2 minutes ago       2 minutes ago

[Vad1mo]

Unfortunately this is the correct behavior. Secrets are immutable. That is why they are recreated on each start. One could of corse check if the last one is still valid and if so reuse it but this is not what the dfple does.