search

Kong / kong-plugin-acme

Let's Encrypt and ACMEv2 integration with Kong - this plugin has been moved into https://github.com/Kong/kong, please open issues and PRs in that repo
Apache License 2.0
36 stars 10 forks source link

Letsencrypt http-01 verification fails due to 404 #10

Closed guyromb closed 4 years ago

guyromb commented 4 years ago

10.0.0.35 - - [15/Feb/2020:17:29:11 +0000] "GET /.well-known/acme-challenge/qgKUJAHBSeiQ6Tq3jYhMBo-Yg3u9rkKAQdXvINoXEsE HTTP/1.1" 404 48 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

Letsencrypt try to access the acme-challenge path but this returns 404 from konga. Any idea how to fix this? Do I need to set the routes manually for this plugin to work?

It is installed globally with basic configurations (Kong 2.0.1).

{
  "created_at": 1581790887,
  "config": {
    "storage_config": {
      "redis": {
        "auth": null,
        "port": null,
        "database": null,
        "host": null
      },
      "shm": {
        "shm_name": "kong"
      },
      "vault": {
        "host": null,
        "port": null,
        "token": null,
        "timeout": null,
        "https": true,
        "kv_path": null
      },
      "kong": {},
      "consul": {
        "host": null,
        "port": null,
        "token": null,
        "timeout": null,
        "https": true,
        "kv_path": null
      }
    },
    "cert_type": "rsa",
    "tos_accepted": true,
    "storage": "kong",
    "domains": [
      "*.DOMAIN.dev",
      "DOMAIN.dev"
    ],
    "api_uri": "https://acme-v02.api.letsencrypt.org",
    "account_email": "EMAIL@EMAIL.COM",
    "renew_threshold_days": 14
  },
  "id": "XXXX",
  "service": null,
  "enabled": true,
  "protocols": [
    "grpc",
    "grpcs",
    "http",
    "https"
  ],
  "name": "acme",
  "consumer": null,
  "route": null,
  "tags": null
}

Screenshot-2020-02-16-at-11-48-45

fffonion commented 4 years ago

Hi, this might be bug or expected behaviour in different context. Could you also share the error log before and after 1 minute of that time please?

UNOPARATOR commented 4 years ago

I have (more or less) the same issue. I'm running kong 2.0.1 along with konga in AKS with kong-admin api using nginx ingress (kong ports 8000 & 8443 are public ingress to 80 & 443 while kong admin api ports 8001 & 8444 & konga's port are private ingress). The plugin is set to redis as storage and I can see the key/values generated. But only the default kong localhost certificate is returned.

Here are my logs:

2020/02/27 09:36:28 [info] 23#0: *81657 client closed connection while SSL handshaking, client: 10.0.6.4, server: 0.0.0.0:8443
2020/02/27 09:36:28 [info] 25#0: *81658 client closed connection while waiting for request, client: 10.0.6.4, server: 0.0.0.0:8000
2020/02/27 09:36:29 [info] 23#0: *81670 client closed connection while SSL handshaking, client: 10.0.7.2, server: 0.0.0.0:8443
2020/02/27 09:36:29 [info] 23#0: *81669 client closed connection while waiting for request, client: 10.0.7.2, server: 0.0.0.0:8000
2020/02/27 09:36:29 [debug] 23#0: *81834 [lua] init.lua:280: [cluster_events] polling events from: 1582792708.097
10.0.7.2 - - [27/Feb/2020:09:36:31 +0000] "GET /status HTTP/1.1" 200 1245 "-" "kube-probe/1.15"
2020/02/27 09:36:34 [info] 25#0: *81803 client closed connection while SSL handshaking, client: 10.0.6.4, server: 0.0.0.0:8443
2020/02/27 09:36:34 [info] 26#0: *81804 client closed connection while waiting for request, client: 10.0.6.4, server: 0.0.0.0:8000
2020/02/27 09:36:34 [debug] 23#0: *81961 [lua] init.lua:280: [cluster_events] polling events from: 1582792708.097
2020/02/27 09:36:35 [info] 26#0: *81817 client closed connection while waiting for request, client: 10.0.7.2, server: 0.0.0.0:8000
2020/02/27 09:36:35 [info] 23#0: *81816 client closed connection while SSL handshaking, client: 10.0.7.2, server: 0.0.0.0:8443
10.0.7.2 - - [27/Feb/2020:09:36:35 +0000] "GET /status HTTP/1.1" 200 1245 "-" "kube-probe/1.15"
2020/02/27 09:36:36 [debug] 23#0: *81987 [lua] certificate.lua:24: log(): [ssl] no SNI provided by client, serving default SSL certificate
2020/02/27 09:36:36 [info] 23#0: *79764 client 10.0.6.70 closed keepalive connection
2020/02/27 09:36:36 [debug] 23#0: *82030 [lua] pkey.lua:157: load_pkey(): load key using fmt: *, type: *
2020/02/27 09:36:36 [debug] 23#0: *82030 [lua] pkey.lua:177: load_pkey(): loaded pkey using PEM_read_bio_PrivateKey
10.0.7.2 - - [27/Feb/2020:09:36:36 +0000] "GET / HTTP/2.0" 404 48 "-" "curl/7.64.1"
2020/02/27 09:36:37 [debug] 23#0: *82030 [lua] client.lua:169: jws(): jws payload: {"protected":{"url":"https:\/\/acme-staging-v02.api.letsencrypt.org\/acme\/new-acct","jwk":{"n":"uO7X4t0W_TJNcFNKt0LEsfe1s2aIvajU-tcLT1ON5nzWDTtC3WorWUT88XYrKCPIOyrOtOMzjKuYvZ0fj-rXRtyxUOCym-ZS4qc1nj-fU_LOWB6RNwAoImjoJvQclVeUqvs4sAUQ7_wWb4d-bq7dJo4GETPWn1jDh2rZM4PbTYIb_p3sRUFLFovHsVEFzf16TvP-HbO4niCddN1QiLh7Uq6e1iwFarZrMTuiz8Svvl1YuI7C-t-S9PkS0z5C1z5e2fanMN_UogGq5CcPik7AgjrRlhI--s_gFNNqdj_RQnBqUrhD04DwCPkY4vZHiycU4duNzaO3FoVQ4ZYimNQVRzpdaYJcxfUV6yl6wluDw11NuWV2PEV-5UajHi0i_0no3H8tUN0SUb4BSjl2ltKEm4INFUJt8etjZOAp8e4GTPoFP7OR-cNlChp51GGb0a187UVojt58hl3wfOYSfPWoRgTOeBeUoSB5S2EmJe3FmXk9xCZ0tvKhLs-K72XaoCv8ELVA95_kbGhmfx1AeM0Kqyddh5oqqFgfSXH9lGVqOWyrhhMdEJlSLFd9-9ivMJnI-IwtNu0BmvTOlrCZWmWckyR9dSs5AMbugntcAeE_U6we-3J4ap4WglJy2cKmIZprEaDYvVqvbIKJAoQtxto13IU9-RNE5au9pKZ77tK1Wu8","kty":"RSA","e":"AQAB"},"alg":"RS256","nonce":"0001xGd1lal3kxSXA9iAQTpalPe_Z3ISmka1-MaSdblSC_U"},"payload":{"termsOfServiceAgreed":true,"contact":["mailto:my-changed-email@gmail.com"]}}
2020/02/27 09:36:37 [debug] 23#0: *82030 [lua] client.lua:215: post(): acme request: https://acme-staging-v02.api.letsencrypt.org/acme/new-acct response: {
  "key": {
    "kty": "RSA",
    "n": "uO7X4t0W_TJNcFNKt0LEsfe1s2aIvajU-tcLT1ON5nzWDTtC3WorWUT88XYrKCPIOyrOtOMzjKuYvZ0fj-rXRtyxUOCym-ZS4qc1nj-fU_LOWB6RNwAoImjoJvQclVeUqvs4sAUQ7_wWb4d-bq7dJo4GETPWn1jDh2rZM4PbTYIb_p3sRUFLFovHsVEFzf16TvP-HbO4niCddN1QiLh7Uq6e1iwFarZrMTuiz8Svvl1YuI7C-t-S9PkS0z5C1z5e2fanMN_UogGq5CcPik7AgjrRlhI--s_gFNNqdj_RQnBqUrhD04DwCPkY4vZHiycU4duNzaO3FoVQ4ZYimNQVRzpdaYJcxfUV6yl6wluDw11NuWV2PEV-5UajHi0i_0no3H8tUN0SUb4BSjl2ltKEm4INFUJt8etjZOAp8e4GTPoFP7OR-cNlChp51GGb0a187UVojt58hl3wfOYSfPWoRgTOeBeUoSB5S2EmJe3FmXk9xCZ0tvKhLs-K72XaoCv8ELVA95_kbGhmfx1AeM0Kqyddh5oqqFgfSXH9lGVqOWyrhhMdEJlSLFd9-9ivMJnI-IwtNu0BmvTOlrCZWmWckyR9dSs5AMbugntcAeE_U6we-3J4ap4WglJy2cKmIZprEaDYvVqvbIKJAoQtxto13IU9-RNE5au9pKZ77tK1Wu8",
    "e": "AQAB"
  },
  "contact": [
    "mailto:my-changed-email@gmail.com"
  ],
  "initialIp": "40.114.190.203",
  "createdAt": "2020-02-26T13:50:17Z",
  "status": "valid"
}
2020/02/27 09:36:38 [debug] 23#0: *82030 [lua] client.lua:169: jws(): jws payload: {"protected":{"url":"https:\/\/acme-staging-v02.api.letsencrypt.org\/acme\/new-order","kid":"https:\/\/acme-staging-v02.api.letsencrypt.org\/acme\/acct\/12584316","alg":"RS256","nonce":"0002gB7yVDPk9JDsJWLYc2zpHTJchJGmpN9YcajuP-okCtk"},"payload":{"identifiers":[{"value":"my-subdomain.westeurope.cloudapp.azure.com","type":"dns"}]}}
2020/02/27 09:36:38 [debug] 23#0: *82030 [lua] client.lua:215: post(): acme request: https://acme-staging-v02.api.letsencrypt.org/acme/new-order response: {
  "status": "pending",
  "expires": "2020-03-05T09:36:38.153838634Z",
  "identifiers": [
    {
      "type": "dns",
      "value": "my-subdomain.westeurope.cloudapp.azure.com"
    }
  ],
  "authorizations": [
    "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/40847622"
  ],
  "finalize": "https://acme-staging-v02.api.letsencrypt.org/acme/finalize/12584316/76894217"
}
2020/02/27 09:36:38 [debug] 23#0: *82030 [lua] client.lua:333: order_certificate(): new order: {"identifiers":[{"value":"my-subdomain.westeurope.cloudapp.azure.com","type":"dns"}],"expires":"2020-03-05T09:36:38.153838634Z","finalize":"https:\/\/acme-staging-v02.api.letsencrypt.org\/acme\/finalize\/12584316\/76894217","status":"pending","authorizations":["https:\/\/acme-staging-v02.api.letsencrypt.org\/acme\/authz-v3\/40847622"]}
2020/02/27 09:36:38 [debug] 23#0: *82030 [lua] client.lua:169: jws(): jws payload: {"protected":{"url":"https:\/\/acme-staging-v02.api.letsencrypt.org\/acme\/authz-v3\/40847622","kid":"https:\/\/acme-staging-v02.api.letsencrypt.org\/acme\/acct\/12584316","alg":"RS256","nonce":"0002kMt9EgjjlDCod28Sa0NbXvdEmKL0fJnc-ypF4XVqehQ"}}
2020/02/27 09:36:38 [debug] 23#0: *82030 [lua] client.lua:215: post(): acme request: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/40847622 response: {
  "identifier": {
    "type": "dns",
    "value": "my-subdomain.westeurope.cloudapp.azure.com"
  },
  "status": "pending",
  "expires": "2020-03-05T09:36:38Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/40847622/U93WJg",
      "token": "lG6IH8x6MMnWfxb4M-CPO9a6eBCd6S4d2hHIWZjYxfw"
    },
    {
      "type": "dns-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/40847622/YSWbpg",
      "token": "lG6IH8x6MMnWfxb4M-CPO9a6eBCd6S4d2hHIWZjYxfw"
    },
    {
      "type": "tls-alpn-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/40847622/nZcbMQ",
      "token": "lG6IH8x6MMnWfxb4M-CPO9a6eBCd6S4d2hHIWZjYxfw"
    }
  ]
}
2020/02/27 09:36:38 [debug] 23#0: *82030 [lua] client.lua:365: order_certificate(): register challenge http-01: lG6IH8x6MMnWfxb4M-CPO9a6eBCd6S4d2hHIWZjYxfw
2020/02/27 09:36:38 [debug] 23#0: *82030 [lua] client.lua:169: jws(): jws payload: {"protected":{"url":"https:\/\/acme-staging-v02.api.letsencrypt.org\/acme\/chall-v3\/40847622\/U93WJg","kid":"https:\/\/acme-staging-v02.api.letsencrypt.org\/acme\/acct\/12584316","alg":"RS256","nonce":"0001lfSY4uPKwXKPcu7-NZLxGxzvOwA9w59DCbF0iWcm4Ec"},"payload":{}}
2020/02/27 09:36:38 [debug] 23#0: *82030 [lua] client.lua:215: post(): acme request: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/40847622/U93WJg response: {
  "type": "http-01",
  "status": "pending",
  "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/40847622/U93WJg",
  "token": "lG6IH8x6MMnWfxb4M-CPO9a6eBCd6S4d2hHIWZjYxfw"
}
10.0.6.4 - - [27/Feb/2020:09:36:38 +0000] "GET /.well-known/acme-challenge/lG6IH8x6MMnWfxb4M-CPO9a6eBCd6S4d2hHIWZjYxfw HTTP/1.1" 404 48 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
10.0.7.2 - - [27/Feb/2020:09:36:39 +0000] "GET /.well-known/acme-challenge/lG6IH8x6MMnWfxb4M-CPO9a6eBCd6S4d2hHIWZjYxfw HTTP/1.1" 404 48 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
10.0.6.4 - - [27/Feb/2020:09:36:39 +0000] "GET /.well-known/acme-challenge/lG6IH8x6MMnWfxb4M-CPO9a6eBCd6S4d2hHIWZjYxfw HTTP/1.1" 404 48 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
10.0.7.2 - - [27/Feb/2020:09:36:39 +0000] "GET /.well-known/acme-challenge/lG6IH8x6MMnWfxb4M-CPO9a6eBCd6S4d2hHIWZjYxfw HTTP/1.1" 404 48 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
10.0.6.230 - - [27/Feb/2020:09:36:39 +0000] "GET /status?size=1000 HTTP/1.1" 200 1245 "-" "-"
2020/02/27 09:36:39 [debug] 23#0: *82030 [lua] client.lua:169: jws(): jws payload: {"protected":{"url":"https:\/\/acme-staging-v02.api.letsencrypt.org\/acme\/order\/12584316\/76894217","kid":"https:\/\/acme-staging-v02.api.letsencrypt.org\/acme\/acct\/12584316","alg":"RS256","nonce":"0001FsPAKN1XVbvg37DsgfKze7LBeCfl7G1Pzos73ZKyrhs"}}
2020/02/27 09:36:39 [debug] 26#0: *82157 [lua] init.lua:280: [cluster_events] polling events from: 1582792708.097
2020/02/27 09:36:40 [debug] 23#0: *82030 [lua] client.lua:215: post(): acme request: https://acme-staging-v02.api.letsencrypt.org/acme/order/12584316/76894217 response: {
  "status": "invalid",
  "expires": "2020-03-05T09:36:38Z",
  "identifiers": [
    {
      "type": "dns",
      "value": "my-subdomain.westeurope.cloudapp.azure.com"
    }
  ],
  "authorizations": [
    "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/40847622"
  ],
  "finalize": "https://acme-staging-v02.api.letsencrypt.org/acme/finalize/12584316/76894217"
}
2020/02/27 09:36:40 [debug] 23#0: *82030 [lua] client.lua:387: order_certificate(): check challenge: {"identifiers":[{"value":"my-subdomain.westeurope.cloudapp.azure.com","type":"dns"}],"expires":"2020-03-05T09:36:38Z","finalize":"https:\/\/acme-staging-v02.api.letsencrypt.org\/acme\/finalize\/12584316\/76894217","status":"invalid","authorizations":["https:\/\/acme-staging-v02.api.letsencrypt.org\/acme\/authz-v3\/40847622"]}
2020/02/27 09:36:40 [debug] 23#0: *82030 [lua] client.lua:169: jws(): jws payload: {"protected":{"url":"https:\/\/acme-staging-v02.api.letsencrypt.org\/acme\/authz-v3\/40847622","kid":"https:\/\/acme-staging-v02.api.letsencrypt.org\/acme\/acct\/12584316","alg":"RS256","nonce":"00028cgWmj93J6ThLmWsdARShMopA9FeQtPRvm7FdJNZyNk"}}
2020/02/27 09:36:40 [debug] 23#0: *82030 [lua] client.lua:215: post(): acme request: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/40847622 response: {
  "identifier": {
    "type": "dns",
    "value": "my-subdomain.westeurope.cloudapp.azure.com"
  },
  "status": "invalid",
  "expires": "2020-03-05T09:36:38Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:unauthorized",
        "detail": "Invalid response from http://my-subdomain.westeurope.cloudapp.azure.com/.well-known/acme-challenge/lG6IH8x6MMnWfxb4M-CPO9a6eBCd6S4d2hHIWZjYxfw [xxx.yyy.zzz.xxx]: 404",
        "status": 403
      },
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/40847622/U93WJg",
      "token": "lG6IH8x6MMnWfxb4M-CPO9a6eBCd6S4d2hHIWZjYxfw",
      "validationRecord": [
        {
          "url": "http://my-subdomain.westeurope.cloudapp.azure.com/.well-known/acme-challenge/lG6IH8x6MMnWfxb4M-CPO9a6eBCd6S4d2hHIWZjYxfw",
          "hostname": "my-subdomain.westeurope.cloudapp.azure.com",
          "port": "80",
          "addressesResolved": [
            "xxx.yyy.zzz.xxx"
          ],
          "addressUsed": "xxx.yyy.zzz.xxx"
        }
      ]
    }
  ]
}
2020/02/27 09:36:40 [debug] 23#0: *82030 [lua] client.lua:399: order_certificate(): authorization status: {"validationRecord":[{"port":"80","url":"http:\/\/my-subdomain.westeurope.cloudapp.azure.com\/.well-known\/acme-challenge\/lG6IH8x6MMnWfxb4M-CPO9a6eBCd6S4d2hHIWZjYxfw","addressUsed":"xxx.yyy.zzz.xxx","addressesResolved":["xxx.yyy.zzz.xxx"],"hostname":"my-subdomain.westeurope.cloudapp.azure.com"}],"token":"lG6IH8x6MMnWfxb4M-CPO9a6eBCd6S4d2hHIWZjYxfw","url":"https:\/\/acme-staging-v02.api.letsencrypt.org\/acme\/chall-v3\/40847622\/U93WJg","type":"http-01","status":"invalid","error":{"type":"urn:ietf:params:acme:error:unauthorized","status":403,"detail":"Invalid response from http:\/\/my-subdomain.westeurope.cloudapp.azure.com\/.well-known\/acme-challenge\/lG6IH8x6MMnWfxb4M-CPO9a6eBCd6S4d2hHIWZjYxfw [xxx.yyy.zzz.xxx]: 404"}}
2020/02/27 09:36:40 [error] 23#0: *82030 [kong] handler.lua:104 failed to update certificate: could not create certificate: challenge invalid: http-01: invalid: Invalid response from http://my-subdomain.westeurope.cloudapp.azure.com/.well-known/acme-challenge/lG6IH8x6MMnWfxb4M-CPO9a6eBCd6S4d2hHIWZjYxfw [xxx.yyy.zzz.xxx]: 404, context: ngx.timer, client: 10.0.7.2, server: 0.0.0.0:8443
2020/02/27 09:36:40 [info] 26#0: *81952 client closed connection while SSL handshaking, client: 10.0.6.4, server: 0.0.0.0:8443
2020/02/27 09:36:40 [info] 24#0: *81953 client closed connection while waiting for request, client: 10.0.6.4, server: 0.0.0.0:8000
2020/02/27 09:36:41 [info] 24#0: *81969 client closed connection while SSL handshaking, client: 10.0.7.2, server: 0.0.0.0:8443
2020/02/27 09:36:41 [info] 26#0: *81970 client closed connection while waiting for request, client: 10.0.7.2, server: 0.0.0.0:8000
10.0.6.230 - - [27/Feb/2020:09:36:41 +0000] "GET / HTTP/1.1" 200 8918 "-" "-"
10.0.7.2 - - [27/Feb/2020:09:36:42 +0000] "GET /status HTTP/1.1" 200 1245 "-" "kube-probe/1.15"
2020/02/27 09:36:43 [info] 23#0: *80730 client 10.0.6.70 closed keepalive connection
10.0.7.2 - - [27/Feb/2020:09:36:44 +0000] "GET /status HTTP/1.1" 200 1245 "-" "kube-probe/1.15"
2020/02/27 09:36:45 [debug] 23#0: *82303 [lua] init.lua:280: [cluster_events] polling events from: 1582792708.097
2020/02/27 09:36:46 [info] 23#0: *82192 client closed connection while SSL handshaking, client: 10.0.6.4, server: 0.0.0.0:8443
2020/02/27 09:36:46 [info] 26#0: *82194 client closed connection while waiting for request, client: 10.0.6.4, server: 0.0.0.0:8000
2020/02/27 09:36:47 [info] 24#0: *82207 client closed connection while SSL handshaking, client: 10.0.7.2, server: 0.0.0.0:8443
2020/02/27 09:36:47 [info] 23#0: *82206 client closed connection while waiting for request, client: 10.0.7.2, server: 0.0.0.0:8000
10.0.6.70 - - [27/Feb/2020:09:36:47 +0000] "GET / HTTP/1.1" 200 8918 "-" "-"
10.0.6.104 - - [27/Feb/2020:09:36:48 +0000] "GET /status?size=1000 HTTP/1.1" 200 1245 "-" "-"
2020/02/27 09:36:49 [info] 23#0: *78080 client 10.0.6.104 closed keepalive connection
2020/02/27 09:36:50 [debug] 26#0: *82413 [lua] init.lua:280: [cluster_events] polling events from: 1582792708.097
10.0.7.2 - - [27/Feb/2020:09:36:51 +0000] "GET /status HTTP/1.1" 200 1245 "-" "kube-probe/1.15"
2020/02/27 09:36:52 [info] 23#0: *82326 client closed connection while SSL handshaking, client: 10.0.6.4, server: 0.0.0.0:8443
2020/02/27 09:36:52 [info] 26#0: *82327 client closed connection while waiting for request, client: 10.0.6.4, server: 0.0.0.0:8000
2020/02/27 09:36:53 [info] 23#0: *82339 client closed connection while SSL handshaking, client: 10.0.7.2, server: 0.0.0.0:8443
2020/02/27 09:36:53 [info] 25#0: *82338 client closed connection while waiting for request, client: 10.0.7.2, server: 0.0.0.0:8000
10.0.6.70 - - [27/Feb/2020:09:36:54 +0000] "GET /status?size=1000 HTTP/1.1" 200 1245 "-" "-"
10.0.6.70 - - [27/Feb/2020:09:36:55 +0000] "GET / HTTP/1.1" 200 8918 "-" "-"
10.0.7.2 - - [27/Feb/2020:09:36:55 +0000] "GET /status HTTP/1.1" 200 1245 "-" "kube-probe/1.15"
2020/02/27 09:36:55 [debug] 26#0: *82527 [lua] init.lua:280: [cluster_events] polling events from: 1582792708.097
fffonion commented 4 years ago

@UNOPARATOR Could you share the share the result of following command curl KONG_IP/.well-known/acme-challenge/x -H "host:my-subdomain.westeurope.cloudapp.azure.com" ?

If it outputs Not found then meaning the plugins is correctly configured, there might be something else to dig. If a Kong error no Route matched with those values is returned or an error from your upstream is returned, then meaning it's misconfigured.

UNOPARATOR commented 4 years ago

The response is : {"message":"no Route matched with those values"}

P.S: Could it be related to me using nginx ingress instead of kong ingress?

fffonion commented 4 years ago

@UNOPARATOR Could be, but let's sort out other possibilities first.

Could you share the following:

UNOPARATOR commented 4 years ago

plugin configuration taken from kong-admin-API/plugins:

{
  "created_at":1582712840, 
  "config": {
    "storage_config": {
    "redis": {
      "auth":"blahblahblah", 
      "port":6379, 
      "database":0, 
      "host":"xyz.redis.cache.windows.net"
    }, 
    "shm": { "shm_name": "kong" }, 
    "vault": { "host": null, "port": null, "token": null, "timeout": null, "https": true, "kv_path": null }, 
    "kong": { }, 
    "consul": { "host": null, "port": null, "token": null, "timeout": null, "https": true, "kv_path": null }
    }, 
    "cert_type":"rsa",
    "tos_accepted":true,
    "storage":"redis",
    "domains":["my-subdomain.westeurope.cloudapp.azure.com"],
    "api_uri":"https:\/\/acme-staging-v02.api.letsencrypt.org",
    "account_email":"my-changed-email@gmail.com",
    "renew_threshold_days":14
  }, "id":"eaf431ab-96e7-48a7-b569-91ae436725bc",
  "service":null,
  "enabled":true,
  "protocols":["grpc", "grpcs", "http", "https"],
  "name":"acme",
  "consumer":null,
  "route":null,
  "tags":null
}

If you mean kong routes then no. All the service routes in kong are using only paths none of which are /.

fffonion commented 4 years ago

@UNOPARATOR the plugin conf looks sane. I would suspect the later is the issue. If a request doesn't match any route in Kong, then no plugin will run.

So in order for this plugin to actually run on that validation path, you will need a route to match path /.well-known/acme-challenge/. The route can be associated a dummy service (like localhost:65535). When the validation request comes in, the plugin will terminate the request and it won't go to dummy service.

UNOPARATOR commented 4 years ago

Per your suggestion, I added a service (http://localhost) and a route to that service (/.well-known/acme-challenge/) and after that it returned Not found to the previous curl command. What I don't understand is, why does the plugin not handle this? Is this a requirement for my case alone (using nginx) or did I miss something hidden somewhere mentioning this is needed?

I still get invalid certificate. When I run curl https://my-subdomain.westeurope.cloudapp.azure.com/my-api/index.html -v this shows up:

*   Trying xxx.yyy.zzz.xxx...
* TCP_NODELAY set
* Connected to my-subdomain.westeurope.cloudapp.azure.com (xxx.yyy.zzz.xxx) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

image

After seeing Not found and getting the Fake certificate, I changed the api url to prod (https://acme-v02.api.letsencrypt.org), the result is the same. I don't know if the certificate created by the staging api is the reason for this.

Edit: By the way, I didn't receive any email for confirmation. I guess I should have received one, right?

fffonion commented 4 years ago

@UNOPARATOR This is by design of plugin system in Kong, that a non-matching route will not run a global plugin. It might be the same case for @guyromb as well. I'll add this note to readme.

The screenshot and curl response looks correct. It failed to verify because it's not issued by a valid CA. After chaing to prod and if you are using the same domain, you will need to manually deleted the previous certificate entity in Kong.

Regarding email: no you won't get email for new certificate, it's only for expiry notice (see https://letsencrypt.org/docs/expiration-emails/). Theoritically you can also provide a dummy email address.

UNOPARATOR commented 4 years ago

Thank you very much for your timely assistance. I deleted the existing certificate (from konga); and after my 2nd attempt (as excepted) at a route, it returned a valid certificate. YAY! =)

I see that you've added the necessary information along with curl commands to the read.me.

P.S.: I now understand how the global plugins work, but I still think the dummy service & the /.well- known/acme-challenge/route should be automatically added when enabling the acme plugin.

fffonion commented 4 years ago

@UNOPARATOR 👍

route should be automatically added

Yeah that is a good point, though some user may not find the dummy service needed. For example if your route matches paths[]=/ then the validation path is already included. There's definitely some space for usibility improvements regarding dummy route and service, let me think over it.

UNOPARATOR commented 4 years ago

By the way, can you share a curl command to enable the plugin with storage_config too?

I couldn't succeed in any way that I tried**, so I enabled it by default then changed the value from the PostgreSQL DB's plugins table. ^_^;

** it always returned to me this:

{"message":"schema violation (config.storage_config: expected a record)","name":"schema violation","fields":{"config":{"storage_config":"expected a record"}},"code":2}

P.S.: You might want to reflect your documentation (read.me) changes to https://docs.konghq.com/hub/kong-inc/acme/too.

fffonion commented 4 years ago

@UNOPARATOR You can use the following syntax:

curl localhost:8001/plugins/uuid -X PATCH -d config.storage_config.redis.auth=1

or use json content type instead (see https://docs.konghq.com/1.1.x/admin-api/#supported-content-types).

Yes thanks for reminding me, I will sync that with docs.konghq.com later 👍

fffonion commented 4 years ago

Consider this resolved, please feel free to reopen if you have other questions : )

ani006 commented 4 years ago

Consider this resolved, please feel free to reopen if you have other questions : )

Hey !! i'm using docker container + dbless mode kong. here my kong.yml : plugin

Docker-compose file : kong: container_name: kong_gateway image: kong:latest volumes:

Please help me out of this. Thanks in advance.

fffonion commented 4 years ago

@ani006 for env variable you will need KONG_LUA_SSL_TRUSTED_CERTIFICATE instead of lua_ssl_trusted_certificate. please open a new issue next time instead of appending to a closed issue : )

ani006 commented 4 years ago

@ani006 for env variable you will need KONG_LUA_SSL_TRUSTED_CERTIFICATE instead of lua_ssl_trusted_certificate. please open a new issue next time instead of appending to a closed issue : )

It worked brother!! But as you can see in my docker-compose i've exposed 80 port but i don't want to expose that port. But if i'm doing that it's giving me err like this => 2020/08/13 12:09:16 [error] 22#0: *272 [kong] handler.lua:104 failed to update certificate: could not create certificate: failed to create new order: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/, context: ngx.timer, client: 103.250.164.250, server: 0.0.0.0:8443 is it because of port or something else !

Sorry for appending here. From next time surely i'll do that. ;)

fffonion commented 4 years ago

@ani006 the error meaning you are hitting the rate limit, you can either try with a different domain or different IP. but i would suggest to use the staging environment first (see config.api_uri). this plugin currently only supports http-01 challenge, meaning exposing the 80 port is a requirement.

ani006 commented 4 years ago

Exposing port 80 is bit risky for me ! what would you suggest me other than that ?