acme error on docker-kong

[lukasa1993]

hello i am running kong from official docker :latest and :2.1 trying to set acme plugin i have dummy service and route and when i am trying to invoke domain with curl -k i get this :

 handler.lua:104 failed to update certificate: acme directory request failed: 18: self signed certificate

db is postgres:9 have this KONG_LUA_SSL_TRUSTED_CERTIFICATE: /etc/ssl/certs/ca-certificates.crt in env

it seems that it doesn't like /etc/ssl/certs/ca-certificates.crt i checked file is there…

[fffonion]

Hi @lukasa1993, can you share the full kong.conf or any other environment variables you are passing to the container?

[lukasa1993]

@fffonion

kong:
    image: kong:2.1
    container_name: kong
    environment:
      KONG_PROXY_ACCESS_LOG: /dev/stdout
      KONG_ADMIN_ACCESS_LOG: /dev/stdout
      KONG_PROXY_ERROR_LOG: /dev/stderr
      KONG_ADMIN_ERROR_LOG: /dev/stderr
      KONG_ADMIN_LISTEN: '0.0.0.0:8001'
      KONG_LUA_SSL_TRUSTED_CERTIFICATE: /etc/ssl/certs/ca-certificates.crt
      KONG_CASSANDRA_CONTACT_POINTS: db
      KONG_DATABASE: 1
      KONG_PG_DATABASE: 2
      KONG_PG_HOST: 3
      KONG_PG_USER: 4
      KONG_PG_PASSWORD: 5
    networks:
      - kong
    ports:
      - "80:8000/tcp"
      - "127.0.0.1:8001:8001/tcp"
      - "443:8443/tcp"
      - "127.0.0.1:8444:8444/tcp"
    healthcheck:
      test: ["CMD", "kong", "health"]
      interval: 10s
      timeout: 10s
      retries: 10
    restart: always

this is all i don't have custom kong.conf

i tried :latest as well same error

[fffonion]

thanks for sharing that. now i realize the error is self signed certificate. if it's indeed the KONG_LUA_SSL_TRUSTED_CERTIFICATE not taking effect, the error will be unable to verify local issuer certificate. so my next question will be what's the api_url you set for the acme plugin? is it an internal CA other than let's encrypt?

[lukasa1993]

@fffonion what you mean api_url ? i used exact same samples as in readme i have acme-dummy service and rout CA cert is whatever comes in docker img i haven't put it my self

[fffonion]

@lukasa1993 do you have any other config fields other than those when adding the plugin?

$ curl http://localhost:8001/plugins \
        -d name=acme \
        -d config.account_email=yourname@yourdomain.com \
        -d config.tos_accepted=true \
        -d config.domains[]=my.secret.domains.com \
        -d config.domains[]=my.anoother.secret.domains.com

[lukasa1993]

i changed storage to KONG but not in same command otherwise its same

[lukasa1993]

but i changed storage to kong after it wasn't working already

[fffonion]

emm i can't reproduce that error on my side. could you send me the docker image ID/hash you are using? it can be viewed by running docker images

[fffonion]

please also share the localhost:8001/plugins endpoint output

[lukasa1993]

kong latest 6cd6bc3dc612 6 days ago 146MB kong 2.1 0d03308d3ec7 6 days ago 147MB

{"next":null,"data":[{"created_at":1594839403,"id":"974d46cd-0ae2-4770-a832-2d2d1972ed3e","tags":null,"enabled":true,"protocols":["grpc","grpcs","http","https"],"name":"acme","consumer":null,"service":null,"route":null,"config":{"storage_config":{"redis":{"auth":null,"port":null,"database":null,"host":null},"shm":{"shm_name":"kong"},"vault":{"host":null,"port":null,"token":null,"timeout":null,"https":false,"tls_server_name":null,"kv_path":null,"tls_verify":true},"kong":{},"consul":{"host":null,"port":null,"token":null,"timeout":null,"https":false,"kv_path":null}},"cert_type":"rsa","tos_accepted":true,"storage":"kong","domains":["example.com"],"api_uri":"https:\/\/acme-v02.api.letsencrypt.org\/directory","account_email":"example@gmail.com","renew_threshold_days":14}},{"created_at":1595156003,"id":"fced96c8-dced-46cd-b07c-9338e0efc8f2","tags":null,"enabled":false,"protocols":["grpc","grpcs","http","https"],"name":"basic-auth","consumer":null,"service":null,"route":{"id":"e931b780-fe8b-4c1b-8dda-364554c78321"},"config":{"hide_credentials":true,"anonymous":null}}]}

@fffonion i swapped actual email and domain but rest is exact same

[fffonion]

those looks good to me. thanks! let's try this: create this file under /tmp/a.lua

local http = require "resty.http"

local hc = http.new()
local a, err = hc:request_uri("https://acme-v02.api.letsencrypt.org/directory")

ngx.say(err)
ngx.say(a and a.body)

run

docker run --rm -it -v /tmp/a.lua:/tmp/a.lua kong:2.1 \
 /usr/local/openresty/bin/resty \
 --http-conf "lua_ssl_trusted_certificate /etc/ssl/certs/ca-certificates.crt;" \
/tmp/a.lua

[lukasa1993]

1@2:~$ vim /tmp/a.lua
1@2:~$ docker run --rm -it -v /tmp/a.lua:/tmp/a.lua kong:2.1 \
>  /usr/local/openresty/bin/resty \
>  --http-conf "lua_ssl_trusted_certificate /etc/ssl/certs/ca-certificates.crt;" \
> /tmp/a.lua

{
  "_KAgDpY3LBc": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org"
  },
  "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"
}

@fffonion seems this worked

[lukasa1993]

didn't meant it solve problem :) it just returned something

[fffonion]

@lukasa1993 yeah that at least verifies the environment is sane. btw you are still seeing that error log right, meaning it's not an ephemeral thing from let's encrypt side.

[lukasa1993]

@fffonion still there i am trying every day same error

[lukasa1993]

atm i am using letsencypt cert from certbot on same server and domain without issue

[fffonion]

i still suspect the issue is not related to trusted certificate, as the error message indicates otherwise. okay let's try to see if the network kong lives is behaving correctly: 1) docker exec into the created kong container, and run a wget https://acme-v02.api.letsencrypt.org/directory -O - 2) try use a different network in the docker compose file, probably the default bridge or host network. is there a custom DNS responder in the kong network you are using?

[lukasa1993]

docker exec -it kong bash
bash-5.0$ wget https://acme-v02.api.letsencrypt.org/directory -O -
Connecting to acme-v02.api.letsencrypt.org (172.65.32.248:443)
writing to stdout
{
  "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org"
  },
  "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert",
  "xPXvcw5Zpwc": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417"
-                    100% |******************************************************************|   658  0:00:00 ETA
written to stdout
bash-5.0$

@fffonion i use bridge network no dns setup that i have done its all default

[lukasa1993]

umm it worked just now for no apparent reason :(

[fffonion]

emmm okay this is really weird

[lukasa1993]

@fffonion issue came back after restart… no idea what fixed it and why its back

[fffonion]

@lukasa1993 I would still put my 5 cents on the network/environment you are running but not plugin itself. Since the let's encrypt API is not likely a self-signed certificate. But it could also be some deeper bug from connection reuse or something.

I have a super hacky idea for debugging this 😂 on kong/plugins/acme/client.lua before Line 93:

  local err = acme_client:init()
  if err then
     -- add following
    local http = require "resty.http"
    local httpc = http.new()
    local res, err = httpc:request_uri("https://acme-v02.api.letsencrypt.org/directory", {
        method = "GET",
        ssl_verify = false,
    })
    ngx.log(ngx.ERR, "response without ssl_verify: ", res and res.body, " error: ", err)

    return nil, nil, err
  end

you can modify the file in host and bind mount into container, try to capture the error log when you saw the error again.

[fffonion]

Closing for inactivity, @lukasa1993 feel free to reopen or create a new issue if you are still seeing issues.

[lukasa1993]

Closing for inactivity, @lukasa1993 feel free to reopen or create a new issue if you are still seeing issues.

hard to replicate still chasing when that happens seems random