Cannot list imported keys due to 502

[cniackz]

Bug describtion

We can create, delete, see status, but when it comes to list imported keys we just get 502 bad gateway. This is on the KES API.

Expected behavior

We should get the key(s).

Alternative:

We can use Vault pod to get those keys listed, that way it works:

/ # vault kv get kv/my-minio/some-key-name-8
========= Data =========
Key                Value
---                -----
some-key-name-8    {"version":"v1","bytes":"1Xrfm4XjvL7MIdv9/H48uso5G0qZBNRJcy5iBqzQhG4=","created_at":"2023-03-14T23:42:00.729167884Z","created_by":"0395d273b16e28073c8004fc195d2411928e80e1d6aea65a50fe36094169c2c4"}

Additional context

1. What version of Go are you using (go version)?: 1.18 as the go.mod in this repo.

2. What operating system and processor architecture are you using (go env)?:

Linux kes-tenant-pool-0-0 5.15.49-linuxkit #1 SMP PREEMPT Tue Sep 13 07:51:32 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

4. Anything else that is important?: Yes, please take a look at this wiki: https://github.com/cniackz/public/wiki/How-to-import-key-in-KES, the important part is that we can perform many actions in the API except listing, and we are using latest KES Version available: 2023-02-15T14-54-37Z but this same issue can be reproduced in older versions like v0.22.3 or even v0.17.x; so maybe not new issue or maybe config issue but can't find the solution:

Additional info:

• MinIO pods are down, so installed Ubuntu pod for communication

• Have all certs in this location below that comes from our secrets in k8s and can communicate via curl

root@ubuntu:~/four-files# cd /root/four-files
root@ubuntu:~/four-files# ls
client.crt  client.key  private.key  public.crt

• Can create a key

curl -v -sSL -k --tlsv1.3 \
    --key client.key \
    --cert client.crt \
    --request POST \
    'https://10.244.1.9:7373/v1/key/create/some-key-name-5'

As a result:

< HTTP/1.1 200 OK
< Date: Tue, 14 Mar 2023 23:27:06 GMT
< Content-Length: 0
< 
* Connection #0 to host 10.244.1.9 left intact

• Can import a key:

curl -v -sSL -k --tlsv1.3 \
    --key client.key \
    --cert client.crt \
    --data '{"bytes":"1Xrfm4XjvL7MIdv9/H48uso5G0qZBNRJcy5iBqzQhG4="}' \
    -X POST 'https://10.244.1.9:7373/v1/key/import/some-key-name-a'

As a result

< HTTP/1.1 200 OK
< Date: Wed, 15 Mar 2023 00:16:50 GMT
< Content-Length: 0
< 
* Connection #0 to host 10.244.1.9 left intact

• Can get status:

curl -v -sSL -k --tlsv1.3 \
    --key client.key \
    --cert client.crt \
    --request GET \
    'https://10.244.1.9:7373/v1/status'

As a result:

< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Wed, 15 Mar 2023 00:17:29 GMT
< Content-Length: 163
< 
{"version":"2023-02-15T14-54-37Z","os":"linux","arch":"arm64","uptime":3218000000000,"num_cpu":5,"num_cpu_used":5,"mem_heap_used":2741392,"mem_stack_used":884736}
* Connection #0 to host 10.244.1.9 left intact

• But can't list and tried several combinations for the search string (*) or even going key by key is same issue:

curl -v -sSL -k --tlsv1.0 \
    --key client.key \
    --cert client.crt \
    --request GET \
    'https://10.244.1.9:7373/v1/key/list/*'

As a result 502:

< HTTP/2 502 
< content-type: application/json; charset=utf-8
< x-content-type-options: nosniff
< content-length: 46
< date: Tue, 14 Mar 2023 23:17:13 GMT
< 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection #0 to host 10.244.4.12 left intact
{"message":"bad gateway: failed to list keys"}

[aead]

This is resolved by now, right? Otherwise, reopen.