Problems with keys greater than 511 in declarative configuration

EdsonLuisVieira commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues

Kong version 3.1.1

kong 3.1.1

Current Behavior

we are upgrading Kong from 2.x to Kong 3.x, and we are facing a blocking bug :cry: We use Kong Databaseless, and with the new LMDB mechanisms from Kong 3.x, our configurations are not loading.

In the tests in declarative config file the length of a consumer's API Key is causing it (but other places may also have the same problem)

When the value has more than 511 characters Kong starts the process, but with empty configurations, also Kong reports: 2023/02/02 14:21:35 [crit] 1129#0: 8 [lua] init.lua:697: init_worker(): worker initialization error: failed to load declarative config file: MDB_BAD_VALSIZE: Unsupported size of key/DB name/data, or wrong DUPFIXED size; this node must be restarted, context: init_worker_by_lua

Expected Behavior

kong in version 3.1.1 can load declarative configuration without errors

Steps To Reproduce

Using the image kong:3.1.1
Load this configuration file bellow: (following the steps in the image readme)
```
---
_format_version: "1.1"
```

consumers:

username: consumer-with-credential-bigger-than-512-chars keyauth_credentials:
- key: 6y8fivuu6jj822b77hcuxp6z33pxgie3t5ypx8mwuxne59gp3yvgt62m2vazuu8b3ahyjhzdmri8bk768qpdp745ijt362f3z5nnwgn785iakpwia5pegwpxki4k8xjzrac2nbg2ff3adrbbgmw7ihj8mj4itwmjaxkkrany5qnb7za46imhc9vwqmg4vpqf5vkza5dfgkr8fphpfrk6bcgz2mxuqgvcvadjb974jpjuctfjptw4j5izb68pywg9a7h85wfdddxh7j7j9yd2cxtvyk7n4cggibec9qtjhci8rdhuwcf2ixtfk4vm6hzj5j6ece9xwuxicdtrizwgm6mrx99av4ukw4c9qp7yr23ij9tutph8gek698765ydjrmgkm88pyzuq7jabpeqe7ae9j8qiew2d9rjdmmd27nmtkdgajrg78347dmrbgvxzjw3f4ewakckxnia3izv7itqa9bufnjn7rd9mwkvzfmuba4hrwk49f2ktzw9npw94

services:

name: httpbin url: https://httpbin.org/anything

routes:

name: httpbin service: httpbin paths:
- / plugins:
- name: key-auth config: key_names:
  - X-API-Key
```
3. see in the log the following error in loading the configuration:
```
    2023/02/02 16:40:15 [error] 1127#0: *889 [kong] config.lua:124 failed loading declarative config into cache: MDB_BAD_VALSIZE: Unsupported size of key/DB name/data, or wrong DUPFIXED size, client: 172.17.0.1, server: kong_admin, request: "POST /config HTTP/1.1", host: "localhost:8001"



### Anything else?

_No response_

chronolaw commented 1 year ago

The default max key size is 511, document of LMDB(https://lmdb.readthedocs.io/en/release/) says:

By default record keys are limited to 511 bytes in length, however this can be adjusted by rebuilding the library. The compile-time key length can be queried via [Environment.max_key_size()](https://lmdb.readthedocs.io/en/release/#lmdb.Environment.max_key_size).

carnei-ro commented 1 year ago

@chronolaw will Kong increase the default value in next versions? Or 511 is the new “hard limit” for pre-built release images?

chronolaw commented 1 year ago

We will have an internal discussion about it. Thanks for your report again.

Ticket: KAG-560.

ghost commented 1 year ago

Are there any updates? I'm willing to try to solve this problem, I just want to confirm: does it make sense to allow max key size over 512? In what scenario would a key size larger than 512 be required? @EdsonLuisVieira

I can guess some random key generation scenarios to prevent key conflicts, but I think the default key size (512) is sufficient in this scenario.

carnei-ro commented 1 year ago

@sabertobihwy In my opinion, an API Key with max of 511 chars is pretty reasonable. The only thing is: I can not upgrade Kong "as is" (without changing this secret in all my clients and reconfiguring Kong), also, it is not documented;

What also made me uncomfortable was the fact: the length of the value of the attribute key is bigger than 511 chars... so, could I not use strings bigger than 511 chars in other places, such as configs for "post function" plugin?! - The fact is I can; It only complains with API Keys.

EdsonLuisVieira commented 1 year ago

hello @sabertobihwy, in our case we are upgrading from version 2.x to 3.x and we can't make the update transparently("not counting deprecations and plugin updates"), but this kind of problem turns on a certain impediment, our main concern is that the size in the yaml is for a value but the error presents it as a key, so we are concerned that this cannot affect other values in custom plugins or definitions of routes and services.

Another point that is relevant is that the command kong config parse show the yaml as valid, so we only get the error when trying to load the new configuration in kong

hanshuebner commented 1 year ago

This will be fixed in the lmdb library: https://github.com/Kong/lua-resty-lmdb/pull/27

Kong / kong