no resolver defined to resolve ocsp.int-x3.letsencrypt.org while requesting certificate status

[Redsandro]

There seems to be an error with the DNS resolver every now and then, when using nginx-proxy together with docker-letsencrypt-nginx-proxy-companion. Nginx will say this:

2019/03/21 15:48:15 [warn] 9#9: no resolver defined to resolve ocsp.int-x3.letsencrypt.org while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org, certificate: "/etc/nginx/certs/*******.com.crt"

And the https node will stay offline (status 502). I deploy again, and then it goes fine.

Unfortunately this is not easily reproducible. It seems to happen randomly. Sometimes everything works as expected, and sometimes I need to deploy two or three times before this error disappears and the upstream node will become available.

I have initially reported this at https://github.com/JrCs/docker-letsencrypt-nginx-proxy-companion/issues/524

Today I received this from a different app on deploy:

2019/03/27 18:06:43 [warn] 8#8: no resolver defined to resolve ocsp.int-x3.letsencrypt.org while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org, certificate: "/etc/nginx/certs/<VIRTUAL_HOST>.crt"
2019/03/27 18:06:46 [error] 8#8: *1 connect() failed (113: No route to host) while connecting to upstream, client: <SERVER_PUBLIC_IP>, server: <VIRTUAL_HOST>, request: "GET / HTTP/2.0", upstream: "<VIRTUAL_HOST_IP>", host: "<VIRTUAL_HOST>"

I am not sure the second message is always there.

Here is the abridged docker-compose.yml for the latter:

version: "3.5"

services:
    dashboard:
        image: redsandro/dashboard
        container_name: rs-dashboard
        depends_on:
            - nginx
            - nginx-gen
            - nginx-letsencrypt
        networks:
            - proxy
        environment:
            VIRTUAL_HOST: dashboard.example.com
            VIRTUAL_PORT: 8080
            LETSENCRYPT_HOST: dashboard.example.com
            LETSENCRYPT_EMAIL: email@example.com

    api:
        image: redsandro/api
        container_name: rs-api
        depends_on:
            - db
        networks:
            - proxy
        ports:
            - '25:25'
        environment:
            VIRTUAL_HOST: api.example.com
            VIRTUAL_PORT: 9080
            LETSENCRYPT_HOST: api.example.com
            LETSENCRYPT_EMAIL: email@example.com

    db:
        image: mongo:4.0
        container_name: rs-db
        restart: always

    nginx:
        image: nginx
        container_name: nginx
        restart: always
        networks:
            - proxy
        ports:
            - 80:80
            - 443:443
        volumes:
            - conf.d:/etc/nginx/conf.d
            - vhost.d:/etc/nginx/vhost.d
            - html:/usr/share/nginx/html
            - certs:/etc/nginx/certs
    nginx-gen:
        image: jwilder/docker-gen
        container_name: nginx-gen
        command: -notify-sighup nginx -watch -wait 5s:30s /etc/docker-gen/templates/nginx.tmpl /etc/nginx/conf.d/default.conf
        restart: always
        networks:
            - proxy
        volumes:
            - conf.d:/etc/nginx/conf.d
            - vhost.d:/etc/nginx/vhost.d
            - html:/usr/share/nginx/html
            - certs:/etc/nginx/certs
            - ./nginx.tmpl:/etc/docker-gen/templates/nginx.tmpl:ro
            - /var/run/docker.sock:/tmp/docker.sock:ro
    nginx-letsencrypt:
        image: jrcs/letsencrypt-nginx-proxy-companion
        container_name: nginx-letsencrypt
        restart: unless-stopped
        volumes:
            - conf.d:/etc/nginx/conf.d
            - vhost.d:/etc/nginx/vhost.d
            - html:/usr/share/nginx/html
            - certs:/etc/nginx/certs
            - /var/run/docker.sock:/var/run/docker.sock:ro
        environment:
            NGINX_DOCKER_GEN_CONTAINER: nginx-gen
            NGINX_PROXY_CONTAINER: nginx

networks:
    proxy:
        external:
            name: proxy

volumes:
    conf.d:
    vhost.d:
    html:
    certs:

[GerardHenninger]

Same issue here, I'll be following this.

[Redsandro]

When docker-compose up -d throws this message, any GET request will cause 502. Only a full down and up seems to work.

In short:

• docker-compose up -d -> GET / -> 502
• docker-compose restart -> GET / -> 502
• docker-compose down && docker-compose up -d -> GET / -> 200

In long:

docker-compose up -d

[warn] 8#8: no resolver defined to resolve ocsp.int-x3.letsencrypt.org while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org, certificate: "/etc/nginx/certs/******.crt"

GET /

[error] 8#8: *1 connect() failed (113: No route to host) while connecting to upstream, client: xx.xx.xx.xx, server: <VIRTUAL_HOST>, request: "GET / HTTP/2.0", upstream: "http://<DOCKER_IP>:8080/", host: "<VIRTUAL_HOST>", referrer: "https://<VIRTUAL_HOST>/"

---

docker-compose restart - Exactly as above. Concequent GET requests not working.

---

docker-compose down && docker-compose up -d

Creating/renewal ds-app.easyterra.com certificates... (ds-app.easyterra.com)
INFO:simp_le:1564: Certificates already exist and renewal is not necessary, exiting with status code 1.

GET /

<VIRTUAL_HOST> xx.xx.xx.xx - - "GET / HTTP/2.0" 200 2210 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3710.0 Safari/537.36"

This one is successful.

Workaround

So the 'workaround' seems to be to break down and recreate the entire stack every time you want to deploy something. This of course is not ideal, as the application will be offline for the time it takes to rebuild every service.

[areaeuro]

I had a similar issue the first time I used the proxy and dockergen as separate containers, which is what you are doing.

Have you tried adding a line to the nginx.tmpl file to tell nginx to use a public nameserver. I added the following line in the server section after line 207: resolver 8.8.8.8;

[Redsandro]

Hi @areaeuro,

I have not tried this. I believe you can just set the resolver in the docker-compose environment variable RESOLVERS, and you do not have to change your template. It will be rendered in this part:

{{ if $.Env.RESOLVERS }}
resolver {{ $.Env.RESOLVERS }};
{{ end }}

I will try next time I observe these errors.

[MarkErik]

Dear all, I am also seeing this (have a 3 container solution - Nginx-web, Nginx-gen, and lets encrypt companion), was wondering if there was a newer solution than the workaround proposed by @Redsandro where we have to bring down and up the containers.

[dm17]

Hi @areaeuro,

I have not tried this. I believe you can just set the resolver in the docker-compose environment variable RESOLVERS, and you do not have to change your template. It will be rendered in this part:

{{ if $.Env.RESOLVERS }}
resolver {{ $.Env.RESOLVERS }};
{{ end }}

I will try next time I observe these errors.

This automated / docker way of using nginx makes debugging a lot more tedious because I can't just read the nginx docs... How do you set the RESEOLVERS env var? Maybe that's the problem? (I have the same issue with the 3 container config.)

[dm17]

When docker-compose up -d throws this message, any GET request will cause 502. Only a full down and up seems to work.

In short:

• docker-compose up -d -> GET / -> 502
• docker-compose restart -> GET / -> 502
• docker-compose down && docker-compose up -d -> GET / -> 200

In long:

docker-compose up -d

[warn] 8#8: no resolver defined to resolve ocsp.int-x3.letsencrypt.org while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org, certificate: "/etc/nginx/certs/******.crt"

GET /

[error] 8#8: *1 connect() failed (113: No route to host) while connecting to upstream, client: xx.xx.xx.xx, server: <VIRTUAL_HOST>, request: "GET / HTTP/2.0", upstream: "http://<DOCKER_IP>:8080/", host: "<VIRTUAL_HOST>", referrer: "https://<VIRTUAL_HOST>/"

docker-compose restart - Exactly as above. Concequent GET requests not working.

docker-compose down && docker-compose up -d

Creating/renewal ds-app.easyterra.com certificates... (ds-app.easyterra.com)
INFO:simp_le:1564: Certificates already exist and renewal is not necessary, exiting with status code 1.

GET /

<VIRTUAL_HOST> xx.xx.xx.xx - - "GET / HTTP/2.0" 200 2210 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3710.0 Safari/537.36"

This one is successful.

Workaround

So the 'workaround' seems to be to break down and recreate the entire stack every time you want to deploy something. This of course is not ideal, as the application will be offline for the time it takes to rebuild every service.

If I do docker-compose down, then docker-compose up (on the 3 container nginx-proxy config), then it does not fix these errors (or they warnings? I still can't tell if they're fatal).

[areaeuro]

Hi @dm17

To answer you first questions, if you study @Redsandro initial post, you will see in the docker file that he is "persisting" the nginx proxy config files, so they are visible to you on the docker host, outside of the container. As to how to set the environment variables, there is also an example in the initial post, under the nginx-letsencryp section of the docker-compose file.

In terms of docker-compose down and docker-compose up not making any changes, this is the way it is supposed to work, to make docker-compose apply the changes you made in the docker.compose.yml file, use the --force-recreate option: docker-compose up --force-recreate If it does not complain, take it down the bring it back up in the background wth the -d option.

I would recommend that you consult the docker-compose documentation if you are unclear of how all this works. https://docs.docker.com/compose/

[letwang]

http {
    resolver 8.8.8.8;
}

[millisami]

Hi @areaeuro,

I have not tried this. I believe you can just set the resolver in the docker-compose environment variable RESOLVERS, and you do not have to change your template. It will be rendered in this part:

{{ if $.Env.RESOLVERS }}
resolver {{ $.Env.RESOLVERS }};
{{ end }}

I will try next time I observe these errors.

Thanks, this did the trick and now the similar error and downtime I was getting is solved.

[fliespl]

But shouldn't RESOLVERS env variable be set at entrypoint according to the codebase based on /etc/resolv.conf?

On start i don't see it in env, but once I execute command locally on docker container it's set.

RESOLVERS=127.0.0.11

    # Compute the DNS resolvers for use in the templates - if the IP contains ":", it's IPv6 and must be enclosed in []
    RESOLVERS=$(awk '$1 == "nameserver" {print ($2 ~ ":")? "["$2"]": $2}' ORS=' ' /etc/resolv.conf | sed 's/ *$//g'); export RESOLVERS

    SCOPED_IPV6_REGEX='\[fe80:(:[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}\]'

    if [[ -z ${RESOLVERS} ]]; then
        echo 'Warning: unable to determine DNS resolvers for nginx' >&2
        unset RESOLVERS
    elif [[ ${RESOLVERS} =~ ${SCOPED_IPV6_REGEX} ]]; then
        echo -n 'Warning: Scoped IPv6 addresses removed from resolvers: ' >&2
        echo "${RESOLVERS}" | grep -Eo "$SCOPED_IPV6_REGEX" | paste -s -d ' ' >&2
        RESOLVERS=$(echo "${RESOLVERS}" | sed -r "s/${SCOPED_IPV6_REGEX}//g" | xargs echo -n); export RESOLVERS
    fi

I am not modyfing CMD so it should go into this piece of code. Anyone?