search

Kong / kong

🦍 The Cloud-Native API Gateway and AI Gateway.
https://konghq.com/install/#kong-community
Apache License 2.0
39.24k stars 4.81k forks source link

Can't start postgresql 16 over tls #12702

Closed michael-todorovic closed 5 months ago

michael-todorovic commented 8 months ago

Is there an existing issue for this?

Kong version ($ kong version)

3.6.1

Current Behavior

I'm currently migrating from pg 13 to 16. Kong 3.6.1 (docker image) worked well on pg13 but when I try to start or just run a kong migrations list on pg16 (bypassing pgbouncer to reduce potential issues), I get:

2024/03/07 15:39:21 [verbose] preparing nginx prefix directory at /usr/local/kong
2024/03/07 15:39:21 [verbose] SSL enabled on admin_gui, no custom certificate set: using default certificates
2024/03/07 15:39:21 [verbose] generating admin_gui SSL certificate (/usr/local/kong/ssl/admin-gui-kong-default.c2024/03/07 15:39:21 [warn] 1298#0: *2 [lua] nginx.lua:300: get_ngx_ssl_from_socket_ctx(): note resty.openssl.auxiliary.nginx is using plain FFI and it's only intended to be used in development, consider using lua-resty-openssl.aux-module in production., context: ngx.timer
Error: 
/usr/local/share/lua/5.1/pgmoon/init.lua:398: attempt to index local 'ssl' (a nil value)
stack traceback:
    /usr/local/share/lua/5.1/pgmoon/init.lua:398: in function 'auth'
    /usr/local/share/lua/5.1/pgmoon/init.lua:268: in function 'connect'
    .../share/lua/5.1/kong/db/strategies/postgres/connector.lua:215: in function 'connect'
    .../share/lua/5.1/kong/db/strategies/postgres/connector.lua:546: in function 'query'
    .../share/lua/5.1/kong/db/strategies/postgres/connector.lua:296: in function 'init'
    /usr/local/share/lua/5.1/kong/db/init.lua:144: in function 'init_connector'
    /usr/local/share/lua/5.1/kong/cmd/migrations.lua:101: in function 'cmd_exec'
    /usr/local/share/lua/5.1/kong/cmd/init.lua:31: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:31>
    [C]: in function 'xpcall'
    /usr/local/share/lua/5.1/kong/cmd/init.lua:31: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:15>
    (command line -e):7: in function 'inline_gen'
    init_worker_by_lua(nginx.conf:136):44: in function <init_worker_by_lua(nginx.conf:136):43>
    [C]: in function 'xpcall'
    init_worker_by_lua(nginx.conf:136):52: in function <init_worker_by_lua(nginx.conf:136):50>
rt) and key (/usr/local/kong/ssl/admin-gui-kong-default.key) for listener
2024/03/07 15:39:21 [verbose] generating admin_gui SSL certificate (/usr/local/kong/ssl/admin-gui-kong-default-ecdsa.crt) and key (/usr/local/kong/ssl/admin-gui-kong-default-ecdsa.key) for listener
2024/03/07 15:39:21 [verbose] generating trusted certs combined file in /usr/local/kong/.ca_combined

I'm using postgres 16.2-1.pgdg120+2 on Debian 12. The TLS config itself is ok:

╰ sslscan --starttls-psql master-postgres.domain.com:5431
Version: 2.0.7
OpenSSL 3.0.11 19 Sep 2023

Connected to 10.154.192.7

Testing SSL server master-postgres.domain.com on port 5431 using SNI name master-postgres.domain.com

  SSL/TLS Protocols:
SSLv2     disabled
SSLv3     disabled
TLSv1.0   disabled
TLSv1.1   disabled
TLSv1.2   enabled
TLSv1.3   enabled

  TLS Fallback SCSV:
Server supports TLS Fallback SCSV

  TLS renegotiation:
Session renegotiation not supported

  TLS Compression:
OpenSSL version does not support compression
Rebuild with zlib1g-dev package for zlib support

  Heartbleed:
TLSv1.3 not vulnerable to heartbleed
TLSv1.2 not vulnerable to heartbleed

  Supported Server Cipher(s):
Preferred TLSv1.3  256 bits  TLS_AES_256_GCM_SHA384        Curve P-256 DHE 256
Accepted  TLSv1.3  256 bits  TLS_CHACHA20_POLY1305_SHA256  Curve P-256 DHE 256
Accepted  TLSv1.3  128 bits  TLS_AES_128_GCM_SHA256        Curve P-256 DHE 256
Preferred TLSv1.2  256 bits  ECDHE-RSA-AES256-GCM-SHA384   Curve P-256 DHE 256
Accepted  TLSv1.2  256 bits  DHE-RSA-AES256-GCM-SHA384     DHE 2048 bits
Accepted  TLSv1.2  256 bits  ECDHE-RSA-CHACHA20-POLY1305   Curve P-256 DHE 256
Accepted  TLSv1.2  256 bits  DHE-RSA-CHACHA20-POLY1305     DHE 2048 bits
Accepted  TLSv1.2  256 bits  DHE-RSA-AES256-CCM8           DHE 2048 bits
Accepted  TLSv1.2  256 bits  DHE-RSA-AES256-CCM            DHE 2048 bits
Accepted  TLSv1.2  256 bits  ECDHE-ARIA256-GCM-SHA384      Curve P-256 DHE 256
Accepted  TLSv1.2  256 bits  DHE-RSA-ARIA256-GCM-SHA384    DHE 2048 bits
Accepted  TLSv1.2  128 bits  ECDHE-RSA-AES128-GCM-SHA256   Curve P-256 DHE 256
Accepted  TLSv1.2  128 bits  DHE-RSA-AES128-GCM-SHA256     DHE 2048 bits
Accepted  TLSv1.2  128 bits  DHE-RSA-AES128-CCM8           DHE 2048 bits
Accepted  TLSv1.2  128 bits  DHE-RSA-AES128-CCM            DHE 2048 bits
Accepted  TLSv1.2  128 bits  ECDHE-ARIA128-GCM-SHA256      Curve P-256 DHE 256
Accepted  TLSv1.2  128 bits  DHE-RSA-ARIA128-GCM-SHA256    DHE 2048 bits
Accepted  TLSv1.2  256 bits  ECDHE-RSA-AES256-SHA384       Curve P-256 DHE 256
Accepted  TLSv1.2  256 bits  DHE-RSA-AES256-SHA256         DHE 2048 bits
Accepted  TLSv1.2  256 bits  ECDHE-RSA-CAMELLIA256-SHA384  Curve P-256 DHE 256
Accepted  TLSv1.2  256 bits  DHE-RSA-CAMELLIA256-SHA256    DHE 2048 bits
Accepted  TLSv1.2  128 bits  ECDHE-RSA-AES128-SHA256       Curve P-256 DHE 256
Accepted  TLSv1.2  128 bits  DHE-RSA-AES128-SHA256         DHE 2048 bits
Accepted  TLSv1.2  128 bits  ECDHE-RSA-CAMELLIA128-SHA256  Curve P-256 DHE 256
Accepted  TLSv1.2  128 bits  DHE-RSA-CAMELLIA128-SHA256    DHE 2048 bits
Accepted  TLSv1.2  256 bits  ECDHE-RSA-AES256-SHA          Curve P-256 DHE 256
Accepted  TLSv1.2  256 bits  DHE-RSA-AES256-SHA            DHE 2048 bits
Accepted  TLSv1.2  256 bits  DHE-RSA-CAMELLIA256-SHA       DHE 2048 bits
Accepted  TLSv1.2  128 bits  ECDHE-RSA-AES128-SHA          Curve P-256 DHE 256
Accepted  TLSv1.2  128 bits  DHE-RSA-AES128-SHA            DHE 2048 bits
Accepted  TLSv1.2  128 bits  DHE-RSA-CAMELLIA128-SHA       DHE 2048 bits
Accepted  TLSv1.2  256 bits  AES256-GCM-SHA384            
Accepted  TLSv1.2  256 bits  AES256-CCM8                  
Accepted  TLSv1.2  256 bits  AES256-CCM                   
Accepted  TLSv1.2  256 bits  ARIA256-GCM-SHA384           
Accepted  TLSv1.2  128 bits  AES128-GCM-SHA256            
Accepted  TLSv1.2  128 bits  AES128-CCM8                  
Accepted  TLSv1.2  128 bits  AES128-CCM                   
Accepted  TLSv1.2  128 bits  ARIA128-GCM-SHA256           
Accepted  TLSv1.2  256 bits  AES256-SHA256                
Accepted  TLSv1.2  256 bits  CAMELLIA256-SHA256           
Accepted  TLSv1.2  128 bits  AES128-SHA256                
Accepted  TLSv1.2  128 bits  CAMELLIA128-SHA256           
Accepted  TLSv1.2  256 bits  AES256-SHA                   
Accepted  TLSv1.2  256 bits  CAMELLIA256-SHA              
Accepted  TLSv1.2  128 bits  AES128-SHA                   
Accepted  TLSv1.2  128 bits  CAMELLIA128-SHA              

  Server Key Exchange Group(s):
TLSv1.3  128 bits  secp256r1 (NIST P-256)
TLSv1.2  128 bits  secp256r1 (NIST P-256)

  SSL Certificate:
Signature Algorithm: sha256WithRSAEncryption
RSA Key Strength:    2048

Subject:  patroni-main-xxxx.domain.com
Altnames: DNS:master-postgres.domain.com, DNS:replica-postgres.domain.Com, DNS:patroni-main-xxxx.domain.com
Issuer:   R3

Not valid before: Mar  7 08:54:32 2024 GMT
Not valid after:  Jun  5 08:54:31 2024 GMT

I checked, just in case, the docker image supports those TLS versions+ciphers. I tried:

On each try, this was the same pg16 cluster of course :smile: I tried to open pgmoon lua code but can't really understand what can be an issue nor how to get more details. I also checked what went at the network level with tcpdump/wireshark but nothing was really showing. On postgres side, I don't have any logs about the failure either.

Do you have an idea how to make progress debugging this issue? Thanks for your help!

Expected Behavior

No response

Steps To Reproduce

I run:

auto_explain.log_analyze = 'True' auto_explain.log_format = 'json' auto_explain.log_min_duration = '-1' auto_explain.log_nested_statements = 'True' auto_explain.log_parameter_max_length = '0' cluster_name = 'int-patroni-kong-g2' effective_cache_size = '1957MB' hot_standby = 'on' listen_addresses = '0.0.0.0' log_connections = 'False' log_destination = 'jsonlog' log_directory = '/var/log/patroni' log_disconnections = 'True' log_duration = 'True' log_filename = 'postgresql' log_hostname = 'True' log_lock_waits = 'True' log_min_duration_statement = '0' log_min_error_statement = 'DEBUG1' log_min_messages = 'DEBUG1' log_replication_commands = 'True' log_statement = 'mod' log_temp_files = '204800' log_timezone = 'Etc/UTC' logging_collector = 'True' maintenance_work_mem = '391MB' max_connections = '100' max_locks_per_transaction = '64' max_prepared_transactions = '0' max_replication_slots = '10' max_standby_archive_delay = '30s' max_standby_streaming_delay = '30s' max_wal_senders = '10' max_worker_processes = '8' port = '5431' shared_buffers = '978MB' shared_preload_libraries = 'auto_explain,pg_stat_statements' ssl = 'on' ssl_cert_file = '/etc/company/certs/patroni-main-xxx.domain.com_crt.pem' ssl_key_file = '/etc/company/certs/patroni-main-xxx.domain.com_key.pem' track_commit_timestamp = 'off' unix_socket_group = 'psql-socket' unix_socket_permissions = '504' wal_keep_size = '128MB' wal_level = 'replica' wal_log_hints = 'on' hba_file = '/var/lib/postgresql/16/main/data/pg_hba.conf' ident_file = '/var/lib/postgresql/16/main/data/pg_ident.conf'

recovery.conf

recovery_target = '' recovery_target_lsn = '' recovery_target_name = '' recovery_target_time = '' recovery_target_timeline = 'latest' recovery_target_xid = ''



### Anything else?

_No response_
sgrzemski commented 8 months ago

I bet that's because of the 0.8.25 to 1.2.0 upgrade of resty-openssl mentioned here: https://docs.konghq.com/gateway/changelog/#3600 and introduced here: https://github.com/Kong/kong/pull/12265. That's a bummer. It's sad to see, but Kong's stability with the new releases is surprisingly low and the introduced versions are full of smaller bugs. E.g. I have to stay on 3.4.2, because 3.5.X does not tolerate custom logging format and I cannot upgrade to 3.6.1, because of the SSL issue.

walter-bd commented 7 months ago

I have the same problem but with postgres 15

bungle commented 7 months ago

Yes, in 3.6.0 there was a couple of big bumps on dependencies. OpenSSL and OpenResty. We need to check what is going on. Thank you for reporting, unfortunately we didn't catch this.

bungle commented 6 months ago

Just link some context here: https://github.com/Kong/pgmoon/blob/v1.16.2/pgmoon/init.lua#L397-L398

local ssl = require("resty.openssl.ssl").from_socket(self.sock) -- it seams this line returns `nil`
local server_cert = ssl:get_peer_certificate()

@michael-todorovic, could you modify that line (in most probably /usr/local/share/lua/5.1/pgmoon/init.lua):

local ssl = require("resty.openssl.ssl").from_socket(self.sock) -- it seams this line returns `nil`

to:

local ssl, err = require("resty.openssl.ssl").from_socket(self.sock) -- it seams this line returns `nil`
if err then
  error(err)
end

And report back?

bungle commented 6 months ago

Also @fffonion could you take a look at it. Is the auxilary module compiled with CE? Is it a requirement now? Is there difference in EE shipped pgmoon regards to this?

pmorelli92 commented 6 months ago

Any updates on this? I got the same, working on 3.5 and stopped with 3.6 and 3.6.1. Leaving my trace just in case :) 

KONG_PG_DATABASE=xxx \
KONG_PG_HOST=xxxxx \
KONG_PG_PASSWORD=xxxx \
KONG_PG_PORT=5432 \
KONG_PG_SSL=on \
KONG_PG_SSL_REQUIRED=on \
KONG_PG_SSL_VERSION=tlsv1_3 \
KONG_PG_USER=kong kong migrations up --v

On 3.5:

2024/04/19 10:53:19 [verbose] Kong: 3.5.0
2024/04/19 10:53:19 [verbose] no config file found at /etc/kong/kong.conf
2024/04/19 10:53:19 [verbose] no config file found at /etc/kong.conf
2024/04/19 10:53:19 [verbose] no config file, skip loading
2024/04/19 10:53:19 [verbose] prefix in use: /usr/local/kong
2024/04/19 10:53:19 [verbose] preparing nginx prefix directory at /usr/local/kong
2024/04/19 10:53:19 [verbose] SSL enabled on proxy, no custom certificate set: using default certificates
2024/04/19 10:53:19 [verbose] generating proxy SSL certificate (/usr/local/kong/ssl/kong-default.crt) and key (/usr/local/kong/ssl/kong-default.key) for listener
2024/04/19 10:53:19 [verbose] generating proxy SSL certificate (/usr/local/kong/ssl/kong-default-ecdsa.crt) and key (/usr/local/kong/ssl/kong-default-ecdsa.key) for listener
2024/04/19 10:53:19 [verbose] SSL enabled on admin, no custom certificate set: using default certificates
2024/04/19 10:53:19 [verbose] generating admin SSL certificate (/usr/local/kong/ssl/admin-kong-default.crt) and key (/usr/local/kong/ssl/admin-kong-default.key) for listener
2024/04/19 10:53:19 [verbose] generating admin SSL certificate (/usr/local/kong/ssl/admin-kong-default-ecdsa.crt) and key (/usr/local/kong/ssl/admin-kong-default-ecdsa.key) for listener
2024/04/19 10:53:19 [verbose] SSL enabled on admin_gui, no custom certificate set: using default certificates
2024/04/19 10:53:19 [verbose] generating admin_gui SSL certificate (/usr/local/kong/ssl/admin-gui-kong-default.crt) and key (/usr/local/kong/ssl/admin-gui-kong-default.key) for listener
2024/04/19 10:53:20 [verbose] generating admin_gui SSL certificate (/usr/local/kong/ssl/admin-gui-kong-default-ecdsa.crt) and key (/usr/local/kong/ssl/admin-gui-kong-default-ecdsa.key) for listener
2024/04/19 10:53:20 [verbose] generating trusted certs combined file in /usr/local/kong/.ca_combined
2024/04/19 10:53:20 [warn] 15#0: *2 [lua] nginx.lua:261: get_ngx_ssl_from_socket_ctx(): note resty.openssl.auxiliary.nginx is using plain FFI and it's only intended to be used in development, consider using lua-resty-openssl.aux-module in production., context: ngx.timer
2024/04/19 10:53:20 [verbose] retrieving database schema state...
2024/04/19 10:53:20 [verbose] schema state retrieved
2024/04/19 10:53:20 [verbose] retrieving database schema state...
2024/04/19 10:53:20 [verbose] schema state retrieved
2024/04/19 10:53:20 [info] Database is already up-to-date

On 3.6.1:

2024/04/19 10:51:50 [verbose] Kong: 3.6.1
2024/04/19 10:51:50 [verbose] no config file found at /etc/kong/kong.conf
2024/04/19 10:51:50 [verbose] no config file found at /etc/kong.conf
2024/04/19 10:51:50 [verbose] no config file, skip loading
2024/04/19 10:51:50 [verbose] prefix in use: /usr/local/kong
2024/04/19 10:51:50 [verbose] preparing nginx prefix directory at /usr/local/kong
2024/04/19 10:51:50 [verbose] SSL enabled on proxy, no custom certificate set: using default certificates
2024/04/19 10:51:50 [verbose] proxy SSL certificate found at /usr/local/kong/ssl/kong-default.crt
2024/04/19 10:51:50 [verbose] proxy SSL certificate found at /usr/local/kong/ssl/kong-default-ecdsa.crt
2024/04/19 10:51:50 [verbose] SSL enabled on admin, no custom certificate set: using default certificates
2024/04/19 10:51:50 [verbose] admin SSL certificate found at /usr/local/kong/ssl/admin-kong-default.crt
2024/04/19 10:51:50 [verbose] admin SSL certificate found at /usr/local/kong/ssl/admin-kong-default-ecdsa.crt
2024/04/19 10:51:50 [verbose] SSL enabled on admin_gui, no custom certificate set: using default certificates
2024/04/19 10:51:50 [verbose] admin_gui SSL certificate found at /usr/local/kong/ssl/admin-gui-kong-default.crt
2024/04/19 10:51:50 [verbose] admin_gui SSL certificate found at /usr/local/kong/ssl/admin-gui-kong-default-ecdsa.crt
2024/04/19 10:51:50 [verbose] generating trusted certs combined file in /usr/local/kong/.ca_combined
2024/04/19 10:51:50 [warn] 64#0: *2 [lua] nginx.lua:300: get_ngx_ssl_from_socket_ctx(): note resty.openssl.auxiliary.nginx is using plain FFI and it's only intended to be used in development, consider using lua-resty-openssl.aux-module in production., context: ngx.timer
Error: 
/usr/local/share/lua/5.1/pgmoon/init.lua:398: attempt to index local 'ssl' (a nil value)
stack traceback:
    /usr/local/share/lua/5.1/pgmoon/init.lua:398: in function 'auth'
    /usr/local/share/lua/5.1/pgmoon/init.lua:268: in function 'connect'
    .../share/lua/5.1/kong/db/strategies/postgres/connector.lua:215: in function 'connect'
    .../share/lua/5.1/kong/db/strategies/postgres/connector.lua:546: in function 'query'
    .../share/lua/5.1/kong/db/strategies/postgres/connector.lua:296: in function 'init'
    /usr/local/share/lua/5.1/kong/db/init.lua:144: in function 'init_connector'
    /usr/local/share/lua/5.1/kong/cmd/migrations.lua:101: in function 'cmd_exec'
    /usr/local/share/lua/5.1/kong/cmd/init.lua:31: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:31>
    [C]: in function 'xpcall'
    /usr/local/share/lua/5.1/kong/cmd/init.lua:31: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:15>
    (command line -e):7: in function 'inline_gen'
    init_worker_by_lua(nginx.conf:170):44: in function <init_worker_by_lua(nginx.conf:170):43>
    [C]: in function 'xpcall'
    init_worker_by_lua(nginx.conf:170):52: in function <init_worker_by_lua(nginx.conf:170):50>
fffonion commented 6 months ago

We are working on fix right now : ) Thanks for you patience.

weberpatr commented 6 months ago

We are working on fix right now : ) Thanks for you patience.

Alerady some updates on this topic? We're currently running into the same issue after updating to 3.6.x

fffonion commented 5 months ago

The fix will be shipped with 3.7.x release.

weberpatr commented 5 months ago

And is there already an approximate ETA for 3.7.x ? This issue is a very big blocker on our side at the moment

fffonion commented 5 months ago

We are still working on publishing the artifacts for 3.7.x, they will be released anytime soon. cc @AndyZhang0707

jeremyjpj0916 commented 5 months ago

I started investigating this as well, Seems lua-resty-openssl-aux-module was not baked into the bazel build process?

https://github.com/search?q=repo%3AKong%2Fkong+path%3A%2F%5Ebuild%5C%2F%2F+lua-resty-openssl-aux-module&type=code

To fix it right away some form of this can be added to those who build from source:

######------ START OF lua-resty-openssl-aux-module ------######

RUN git clone --depth 1 -b 0.3.0 --single-branch https://github.com/fffonion/lua-resty-openssl-aux-module \

######------ END OF lua-resty-openssl-aux-module ------######

#### Do your bazel Kong build code here ####

# Post kong build then execute this:

# Additional make install step POST COMPILE per docs: https://github.com/fffonion/lua-resty-openssl-aux-module/tree/0.3.0
RUN cd /lua-resty-openssl-aux-module && make install

Hoping 3.7.x drops soon so I can skip adding this as a fork in my build flow though!

chronolaw commented 5 months ago

kong 3.7 is released now. See: https://github.com/Kong/kong/releases/tag/3.7.0

jeremyjpj0916 commented 5 months ago

kong 3.7 is released now. See: https://github.com/Kong/kong/releases/tag/3.7.0

@chronolaw , not seeing anything about Kong fixing the postgres tls issue though in the changelog? Did the fix make it into the version, trying to see what PR fixed it if the changelog lacks a reference point?

Edit - might be related but says its a temporary fix - https://github.com/Kong/kong/pull/12665 , is the full fix going to be adding lua-resty-openssl-aux-module to the build flow?

Edit edit - yeah seems 1.2.1 bump https://github.com/fffonion/lua-resty-openssl/pull/148/files has the fixes needed for that version of OpenResty that ships with Kong without the need for the aux module necessarily.

chronolaw commented 5 months ago

@fffonion , could you give us the explanation of this fix in 3.7?

weberpatr commented 5 months ago

and when will it be pushed to docker hub? :)

Water-Melon commented 5 months ago

It's fixed in 3.7, please refer to https://github.com/Kong/kong/issues/12592

pmorelli92 commented 5 months ago

and when will it be pushed to docker hub? :)

Bump

weberpatr commented 5 months ago

and when will it be pushed to docker hub? :)

Bump

it's available now, was pushed last evening

pmorelli92 commented 5 months ago

I just installed the new images and I can confirm it works. Thanks for the effort!

weberpatr commented 5 months ago

Can also confirm - issue seems to be solved. Thanks a lot!

gregorgebhardt commented 5 months ago

Thanks a lot for solving this issue with Kong 3.7. We were stuck on Kong 3.5 so far because of this. Now, we are trying to upgrade to Kong 3.7. However, we cannot upgrade from 3.5 to 3.7 directly (see also the table here) as we're getting the error Error: Database has pending migrations; run 'kong migrations finish' in the Kong pre-upgrade-migrations container. How did you upgrade from 3.5 to 3.7 directly?

weberpatr commented 5 months ago

Thanks a lot for solving this issue with Kong 3.7. We were stuck on Kong 3.5 so far because of this. Now, we are trying to upgrade to Kong 3.7. However, we cannot upgrade from 3.5 to 3.7 directly (see also the table here) as we're getting the error Error: Database has pending migrations; run 'kong migrations finish' in the Kong pre-upgrade-migrations container. How did you upgrade from 3.5 to 3.7 directly?

we migrated from 3.5 to 3.7 successfully in one step by running kong migrations up && kong migrations finish

gregorgebhardt commented 5 months ago

Thanks a lot for solving this issue with Kong 3.7. We were stuck on Kong 3.5 so far because of this. Now, we are trying to upgrade to Kong 3.7. However, we cannot upgrade from 3.5 to 3.7 directly (see also the table here) as we're getting the error Error: Database has pending migrations; run 'kong migrations finish' in the Kong pre-upgrade-migrations container. How did you upgrade from 3.5 to 3.7 directly?

we migrated from 3.5 to 3.7 successfully in one step by running kong migrations up && kong migrations finish

Thank you for your help, we will try to do it the same way