Kong crashing with `missing shared dict` when specifying custom config.memory.dictionary_name in proxy-cache plugin

[cristichiru]

Summary

Kong crashing with missing shared dict when specifying custom config.memory.dictionary_name in proxy-cache plugin. Previous versions worked fine. Removing config.memory.dictionary_name from the plugin config works fine, but... I want to use a dedicated dict with a larger size.

Steps To Reproduce

1. Set env: KONG_NGINX_HTTP_LUA_SHARED_DICT=proxy_cache_storage 128m

2. Set the proxy-cache conf: config.memory.dictionary_name=proxy_cache_storage:

   kong.yml: |-
     _format_version: "1.1"
   
     services:
     - name: aservice
       host: somehost.com
       path: /
       port: 443
       protocol: https
       routes:
       - name: aroute
         hosts:
         - myhost.com
         paths:
         - /
         preserve_host: false
         protocols:
         - https
         strip_path: false
         https_redirect_status_code: 301
       plugins:
       - name: proxy-cache
         config:
           strategy: memory
           memory:
             dictionary_name: proxy_cache_storage
           content_type:
           - text/plain
           - application/json
           - application/json;charset=utf-8
           - application/json; charset=utf-8
           #cache_control: true
           #storage_ttl: 1
           cache_ttl: 1
         enabled: true
         protocols:
         - https
         - http

3. Pod env:

   env:
   - name: KONG_ADMIN_ACCESS_LOG
       value: /dev/stdout
   - name: KONG_ADMIN_ERROR_LOG
       value: /dev/stderr
   - name: KONG_ADMIN_GUI_ACCESS_LOG
       value: /dev/stdout
   - name: KONG_ADMIN_GUI_ERROR_LOG
       value: /dev/stderr
   - name: KONG_ADMIN_LISTEN
       value: '0.0.0.0:8001'
   - name: KONG_ANONYMOUS_REPORTS
       value: 'false'
   - name: KONG_CLUSTER_LISTEN
       value: 'off'
   - name: KONG_DATABASE
       value: 'off'
   - name: KONG_DECLARATIVE_CONFIG
       value: /kong_dbless/kong.yml
   - name: KONG_LOG_LEVEL
       value: info
   - name: KONG_LUA_PACKAGE_PATH
       value: /opt/?.lua;/opt/?/init.lua;;
   - name: KONG_MEM_CACHE_SIZE
       value: 2048m
   - name: KONG_NGINX_CLIENT_HEADER_BUFFER_SIZE
       value: 32k
   - name: KONG_NGINX_HTTP_INCLUDE
       value: /kong/servers.conf
   - name: KONG_NGINX_HTTP_LUA_SHARED_DICT
       value: proxy_cache_storage 128m
   - name: KONG_NGINX_HTTP_OUTPUT_BUFFERS
       value: 8 64k
   - name: KONG_NGINX_LARGE_CLIENT_HEADER_BUFFERS
       value: 4 16k
   - name: KONG_NGINX_PROXY_LARGE_CLIENT_HEADER_BUFFERS
       value: 16 128k
   - name: KONG_NGINX_WORKER_PROCESSES
       value: '1'
   - name: KONG_PORTAL_API_ACCESS_LOG
       value: /dev/stdout
   - name: KONG_PORTAL_API_ERROR_LOG
       value: /dev/stderr
   - name: KONG_PORT_MAPS
       value: '80:8000, 443:8443'
   - name: KONG_PREFIX
       value: /kong_prefix/
   - name: KONG_PROXY_ACCESS_LOG
       value: /dev/stdout
   - name: KONG_PROXY_ERROR_LOG
       value: /dev/stderr
   - name: KONG_PROXY_LISTEN
       value: '0.0.0.0:8000, 0.0.0.0:8443 http2 ssl'
   - name: KONG_STATUS_LISTEN
       value: '0.0.0.0:8100'
   - name: KONG_STREAM_LISTEN
       value: '0.0.0.0:5702'
   - name: KONG_NGINX_DAEMON
       value: 'off'

4. Checked the kong container, my lua_shared_dict is there:

   bash-5.0$ grep -n -B2 -A2 proxy_cache_storage /kong_prefix/nginx-kong.conf
   38-include /kong/servers.conf;
   39-lua_shared_dict prometheus_metrics 5m;
   40=lua_shared_dict proxy_cache_storage 128m;
   41-output_buffers 8 64k;
   42-ssl_prefer_server_ciphers off;

5. Error:

   2021/01/05 23:06:42 [error] 1#0: init_by_lua error: /usr/local/share/lua/5.1/kong/init.lua:474: error parsing declarative config file /kong_dbless/kong.yml:
   in 'services':
     - in entry 37 of 'services':
       in 'plugins':
         - in entry 1 of 'plugins':
           in '@entity':
             - in entry 1 of '@entity': missing shared dict 'proxy_cache_storage'
   stack traceback:
       [C]: in function 'error'
       /usr/local/share/lua/5.1/kong/init.lua:474: in function 'init'
       init_by_lua:11: in main chunk

Additional Details & Logs

• Kong version 2.2.1
• Operating system: docker kong/kong:2.2.1 (Alpine Linux)
• Deployment: Kubernetes using the official chart, version: 1.8.0

[bungle]

Hi @cristichiru,

I tried to reproduce this with next using:

KONG_NGINX_HTTP_LUA_SHARED_DICT="proxy_cache_storage 128m" \
KONG_DATABASE=off \
KONG_DECLARATIVE_CONFIG=pcache.yaml \
./bin/kong start --vv
``

And it started fine.

pcache.yaml:

```yml
_format_version: "1.1"
services:
- name: aservice
  host: somehost.com
  path: /
  port: 443
  protocol: https
  routes:
  - name: aroute
    hosts:
    - myhost.com
    paths:
    - /
    preserve_host: false
    protocols:
    - https
    strip_path: false
    https_redirect_status_code: 301
  plugins:
  - name: proxy-cache
    config:
      strategy: memory
      memory:
        dictionary_name: proxy_cache_storage
      content_type:
      - text/plain
      - application/json
      - application/json;charset=utf-8
      - application/json; charset=utf-8
      #cache_control: true
      #storage_ttl: 1
      cache_ttl: 1
    enabled: true
    protocols:
    - https
    - http

And plugin shows correctly:

{
    "data": [
        {
            "config": {
                "cache_control": false,
                "cache_ttl": 1,
                "content_type": [
                    "text/plain",
                    "application/json",
                    "application/json;charset=utf-8",
                    "application/json; charset=utf-8"
                ],
                "memory": {
                    "dictionary_name": "proxy_cache_storage"
                },
                "request_method": [
                    "GET",
                    "HEAD"
                ],
                "response_code": [
                    200,
                    301,
                    404
                ],
                "storage_ttl": null,
                "strategy": "memory",
                "vary_headers": null,
                "vary_query_params": null
            },
            "consumer": null,
            "created_at": 1610055606,
            "enabled": true,
            "id": "456fb5fd-b269-5bfe-b970-bdac3ee1305e",
            "name": "proxy-cache",
            "protocols": [
                "https",
                "http"
            ],
            "route": null,
            "service": {
                "id": "cf0edba6-af0d-5df2-9dfd-53ef4ae9392b"
            },
            "tags": null
        }
    ],
    "next": null
}

I even tried :8001/config endpoint too:

$ http :8001/config config=@pcache.yaml
HTTP/1.1 201 Created
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Length: 1291
Content-Type: application/json; charset=utf-8
Date: Thu, 07 Jan 2021 21:43:32 GMT
Server: kong/2.2.1
X-Kong-Admin-Latency: 531

{
    "plugins": {
        "456fb5fd-b269-5bfe-b970-bdac3ee1305e": {
            "config": {
                "cache_control": false,
                "cache_ttl": 1,
                "content_type": [
                    "text/plain",
                    "application/json",
                    "application/json;charset=utf-8",
                    "application/json; charset=utf-8"
                ],
                "memory": {
                    "dictionary_name": "proxy_cache_storage"
                },
                "request_method": [
                    "GET",
                    "HEAD"
                ],
                "response_code": [
                    200,
                    301,
                    404
                ],
                "strategy": "memory"
            },
            "created_at": 1610055811,
            "enabled": true,
            "id": "456fb5fd-b269-5bfe-b970-bdac3ee1305e",
            "name": "proxy-cache",
            "protocols": [
                "https",
                "http"
            ],
            "service": {
                "id": "cf0edba6-af0d-5df2-9dfd-53ef4ae9392b"
            }
        }
    },
    "routes": {
        "583d2e52-dbfc-561a-9c64-eb4bdd717ad9": {
            "created_at": 1610055811,
            "hosts": [
                "myhost.com"
            ],
            "https_redirect_status_code": 301,
            "id": "583d2e52-dbfc-561a-9c64-eb4bdd717ad9",
            "name": "aroute",
            "path_handling": "v0",
            "paths": [
                "/"
            ],
            "preserve_host": false,
            "protocols": [
                "https"
            ],
            "regex_priority": 0,
            "request_buffering": true,
            "response_buffering": true,
            "service": {
                "id": "cf0edba6-af0d-5df2-9dfd-53ef4ae9392b"
            },
            "strip_path": false,
            "updated_at": 1610055811
        }
    },
    "services": {
        "cf0edba6-af0d-5df2-9dfd-53ef4ae9392b": {
            "connect_timeout": 60000,
            "created_at": 1610055811,
            "host": "somehost.com",
            "id": "cf0edba6-af0d-5df2-9dfd-53ef4ae9392b",
            "name": "aservice",
            "path": "/",
            "port": 443,
            "protocol": "https",
            "read_timeout": 60000,
            "retries": 5,
            "updated_at": 1610055811,
            "write_timeout": 60000
        }
    }
}

So we need more information on how to reproduce. @hbagdi / @rainest, do you remember where and how the validation happens on Kubernetes? @cristichiru are you sure this error is from machine that has that KONG_NGINX_HTTP_LUA_SHARED_DICT="proxy_cache_storage 128m" ?

[rainest]

There isn't anything that'd validate this in K8S tooling: it will just pass whatever environment variables you provide (and that apparently worked, given the content of nginx-kong.conf) and attempt to apply whatever declarative config you provide. All validation of the NGINX config and Kong config is up to them (there's some validation of Kong config when using the ingress controller, but none when providing a declarative config directly, as is the case here).

[cristichiru]

Hi @bungle,

Yes, I am sure it was that machine, as there is only one pod with kong, and the details above were taken directly from the pod runtime.

What else can I provide to help with this?

Thanks.

[cristichiru]

I did exactly what you did in a docker container. Surprisingly, it worked. Now I am confused.

# docker run --rm -it --name docker-kong kong/kong:2.2.1 bash
bash-5.0$ cd /usr/local/kong
bash-5.0$ export KONG_NGINX_HTTP_LUA_SHARED_DICT="proxy_cache_storage 128m"
bash-5.0$ export KONG_DATABASE=off
bash-5.0$ echo '_format_version: "1.1"
services:
- name: aservice
  host: somehost.com
  path: /
  port: 443
  protocol: https
  routes:
  - name: aroute
    hosts:
    - myhost.com
    paths:
    - /
    preserve_host: false
    protocols:
    - https
    strip_path: false
    https_redirect_status_code: 301
  plugins:
  - name: proxy-cache
    config:
      strategy: memory
      memory:
        dictionary_name: proxy_cache_storage
      content_type:
      - text/plain
      - application/json
      - application/json;charset=utf-8
      - application/json; charset=utf-8
      #cache_control: true
      #storage_ttl: 1
      cache_ttl: 1
    enabled: true
    protocols:
    - https
    - http
' > pcache.yaml
bash-5.0$ export KONG_DECLARATIVE_CONFIG=pcache.yaml
bash-5.0$ kong start -vv                <---- success!!!

in another shell, docker exec -it -u 0 docker-kong bash:

bash-5.0# apk add httpie
bash-5.0# http :8001/plugins
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Length: 618
Content-Type: application/json; charset=utf-8
Date: Sat, 09 Jan 2021 02:28:51 GMT
Server: kong/2.2.1
X-Kong-Admin-Latency: 252

{
    "data": [
        {
            "config": {
                "cache_control": false,
                "cache_ttl": 1,
                "content_type": [
                    "text/plain",
                    "application/json",
                    "application/json;charset=utf-8",
                    "application/json; charset=utf-8"
                ],
                "memory": {
                    "dictionary_name": "proxy_cache_storage"
                },
                "request_method": [
                    "GET",
                    "HEAD"
                ],
                "response_code": [
                    200,
                    301,
                    404
                ],
                "storage_ttl": null,
                "strategy": "memory",
                "vary_headers": null,
                "vary_query_params": null
            },
            "consumer": null,
            "created_at": 1610159302,
            "enabled": true,
            "id": "456fb5fd-b269-5bfe-b970-bdac3ee1305e",
            "name": "proxy-cache",
            "protocols": [
                "https",
                "http"
            ],
            "route": null,
            "service": {
                "id": "cf0edba6-af0d-5df2-9dfd-53ef4ae9392b"
            },
            "tags": null
        }
    ],
    "next": null
}

Now I am even more confused...

Going back to the k8s deployment... because of the helm chart and some of the env I have overridden (like KONG_PREFIX), the inside of the k8s container is different though:

bash-5.0$ ls -1 /kong*
/kong:
servers.conf

/kong_dbless:
kong.yml

/kong_prefix:
client_body_temp
fastcgi_temp
logs
nginx-kong-stream.conf
nginx-kong.conf
nginx.conf
pids
proxy_temp
scgi_temp
ssl
stream_config.sock
uwsgi_temp

[cristichiru]

Even more confusing.... trying it directly in the running container in k8s...

docker exec -it -u 0 k8s_proxy_my-kong-75f977787c-4nv88_kong_b1d3120a-239f-4e33-a863-23bb1ab9dfcd_0 bash
bash-5.0# apk add httpie
bash-5.0# echo '_format_version: "1.1"
> services:
> - name: aservice
>   host: somehost.com
>   path: /
>   port: 443
>   protocol: https
>   routes:
>   - name: aroute
>     hosts:
>     - myhost.com
>     paths:
>     - /
>     preserve_host: false
>     protocols:
>     - https
>     strip_path: false
>     https_redirect_status_code: 301
>   plugins:
>   - name: proxy-cache
>     config:
>       strategy: memory
>       memory:
>         dictionary_name: proxy_cache_storage
>       content_type:
>       - text/plain
>       - application/json
>       - application/json;charset=utf-8
>       - application/json; charset=utf-8
>       #cache_control: true
>       #storage_ttl: 1
>       cache_ttl: 1
>     enabled: true
>     protocols:
>     - https
>     - http
> ' > pcache.yaml

bash-5.0# http :8001/config config=@pcache.yaml
HTTP/1.1 201 Created
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Length: 1291
Content-Type: application/json; charset=utf-8
Date: Sat, 09 Jan 2021 02:46:32 GMT
Server: kong/2.2.1
X-Kong-Admin-Latency: 270

{
    "plugins": {
        "456fb5fd-b269-5bfe-b970-bdac3ee1305e": {
            "config": {
                "cache_control": false,
                "cache_ttl": 1,
                "content_type": [
                    "text/plain",
                    "application/json",
                    "application/json;charset=utf-8",
                    "application/json; charset=utf-8"
                ],
                "memory": {
                    "dictionary_name": "proxy_cache_storage"
                },
.......................

however, If I put:

      memory:
        dictionary_name: proxy_cache_storage

in the ConfigMap and restart the POD, will still give the same init error as initially described.

[cristichiru]

Failure

Success

[cristichiru]

Still got this issue in kong 2.4.1 😢

The check is here: https://github.com/Kong/kong-plugin-proxy-cache/blob/559675e2453615eaf14b8ee6fa74b9f10447391f/kong/plugins/proxy-cache/schema.lua#L8

Tried to use another "built-in" dictionary, and... it works?!

Taking a look in nginx-kong.coonf:

## If I set in the cache plugin "kong_core_db_cache_2",  will work, as it seems to be present in 'ngx.shared' object, checked by the plugin code
lua_shared_dict kong_core_db_cache_2        128m;

lua_shared_dict kong_core_db_cache_miss_2   12m;
lua_shared_dict kong_db_cache_2             128m;
lua_shared_dict kong_db_cache_miss_2        12m;

underscores_in_headers on;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GC
M-SHA384;

# injected nginx_http_* directives
client_body_buffer_size 8k;
client_max_body_size 0;

## But the one defined by me, "proxy_cache_storage", a few lines below is not loaded as "it missing"!??
lua_shared_dict proxy_cache_storage  128m;

lua_shared_dict prometheus_metrics 5m;

curl -sS kong-admin:8001/plugins | jq '.data[] | select(.name=="proxy-cache").config'

{
  "request_method": [
    "GET",
    "HEAD"
  ],
  "cache_control": false,
  "vary_query_params": null,
  "cache_ttl": 2,
  "strategy": "memory",
  "response_code": [
    200,
    301,
    404
  ],
  "memory": {
    "dictionary_name": "kong_core_db_cache_2"
  },
  "vary_headers": null,
  "content_type": [
    "text/plain",
    "application/json",
    "application/json;charset=utf-8",
    "application/json; charset=utf-8",
    "text/xml",
    "text/xml; charset=utf-8",
    "text/xml;charset=utf-8"
  ],
  "storage_ttl": null
}

[StarlightIbuki]

Dear contributor, We're closing this issue as there hasn't been any update to it for a long time. If the issue is still relevant in the latest version, please feel free to reopen it. We're more than happy to revisit it again. Your contribution is greatly appreciated! Please have a look at our pledge to the community for more information. Sincerely, Kong Gateway Team