search

k3s-io / k3s

Lightweight Kubernetes
https://k3s.io
Apache License 2.0
27.62k stars 2.31k forks source link

[Release-1.27] - `ListObjectsV2 search parameter metadata not implemented` when using Cloudflare R2 #8930

Closed brandond closed 9 months ago

brandond commented 10 months ago

Backport fix for ListObjectsV2 search parameter metadata not implemented when using Cloudflare R2

VestigeJ commented 9 months ago

Reproduced here - https://github.com/k3s-io/k3s/issues/8924#issuecomment-1823315855

Environment Details

Validated using VERSION=v1.27.8-rc1+k3s1

Infrastructure

Node(s) CPU architecture, OS, and version:

Linux 5.14.21-150500.53-default x86_64 GNU/Linux PRETTY_NAME="SUSE Linux Enterprise Server 15 SP5"

Cluster Configuration:

NAME         STATUS   ROLES                       AGE    VERSION
ip-1-1-2-7   Ready    control-plane,etcd,master   6m3s   v1.27.8-rc1+k3s1

Config.yaml:

write-kubeconfig-mode: 644
debug: true
token: YOUR_TOKEN_HERE
profile: cis
selinux: true
node-external-ip: 1.1.2.7
protect-kernel-defaults: true
cluster-init: true

etcd-s3: true
etcd-s3-bucket: "k3s-etcd-testing"
etcd-s3-endpoint: "epepepepeeep.r2.cloudflarestorage.com"
etcd-s3-access-key: "keykeykeykeyyyy"
etcd-s3-secret-key: "keykeykeykeeeey"

YOUR_REPRODUCED_RESULTS_HERE

``` $ curl https://get.k3s.io --output install-"k3s".sh $ sudo chmod +x install-"k3s".sh $ sudo groupadd --system etcd && sudo useradd -s /sbin/nologin --system -g etcd etcd $ sudo modprobe ip_vs_rr $ sudo modprobe ip_vs_wrr $ sudo modprobe ip_vs_sh $ sudo printf "on_oovm.panic_on_oom=0 \nvm.overcommit_memory=1 \nkernel.panic=10 \nkernel.panic_ps=1 \nkernel.panic_on_oops=1 \n" > ~/90-kubelet.conf $ sudo cp 90-kubelet.conf /etc/sysctl.d/ $ sudo systemctl restart systemd-sysctl $ sudo INSTALL_K3S_VERSION=v1.27.8-rc1+k3s1 INSTALL_K3S_EXEC=server ./install-k3s.sh $ w2 kg no,po -A //watch -n2 kubectl get nodes,pods -A $ sudo /usr/local/bin/k3s etcd-snapshot save //observe r2 bucket for snapshot files ``` **Results:** ![Screenshot 2023-11-27 at 10 31 30 AM](https://github.com/k3s-io/k3s/assets/8961630/7f8aa32a-498f-459f-be0f-5a934e60d106)

$ sudo /usr/local/bin/k3s etcd-snapshot save

WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --token found in config.yaml, skipping
WARN[0000] Unknown flag --profile found in config.yaml, skipping
WARN[0000] Unknown flag --selinux found in config.yaml, skipping
WARN[0000] Unknown flag --node-external-ip found in config.yaml, skipping
WARN[0000] Unknown flag --protect-kernel-defaults found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-init found in config.yaml, skipping
DEBU[0000] Attempting to retrieve extra metadata from k3s-etcd-snapshot-extra-metadata ConfigMap
DEBU[0000] Error encountered attempting to retrieve extra metadata from k3s-etcd-snapshot-extra-metadata ConfigMap, error: configmaps "k3s-etcd-snapshot-extra-metadata" not found
INFO[0000] Saving etcd snapshot to /var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-1-1-2-7-1701109335
{"level":"info","ts":"2023-11-27T18:22:15.067782Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-1-1-2-7-1701109335.part"}
{"level":"info","ts":"2023-11-27T18:22:15.070373Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-11-27T18:22:15.07082Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2023-11-27T18:22:15.103608Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-11-27T18:22:15.119437Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"3.5 MB","took":"now"}
{"level":"info","ts":"2023-11-27T18:22:15.119638Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-1-1-2-7-1701109335"}
INFO[0000] Checking if S3 bucket k3s-etcd-testing exists
INFO[0000] S3 bucket k3s-etcd-testing exists
INFO[0000] Saving etcd snapshot on-demand-ip-1-1-2-7-1701109335 to S3
INFO[0000] Uploading snapshot to s3://k3s-etcd-testing//var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-1-1-2-7-1701109335
INFO[0001] Uploaded snapshot metadata s3://k3s-etcd-testing/.metadata/on-demand-ip-1-1-2-7-1701109335
INFO[0001] S3 upload complete for on-demand-ip-1-1-2-7-1701109335
INFO[0001] Reconciling ETCDSnapshotFile resources
DEBU[0001] Found snapshotFile for on-demand-ip-1-1-2-7-1701109315 with key local-on-demand-ip-1-1-2-7-1701109315
DEBU[0001] Found snapshotFile for on-demand-ip-1-1-2-7-1701109335 with key local-on-demand-ip-1-1-2-7-1701109335
DEBU[0001] Found snapshotFile for on-demand-ip-1-1-2-7-1701109335 with key s3-on-demand-ip-1-1-2-7-1701109335
DEBU[0001] Found ETCDSnapshotFile for on-demand-ip-1-1-2-7-1701109315 with key local-on-demand-ip-1-1-2-7-1701109315
DEBU[0001] Found ETCDSnapshotFile for on-demand-ip-1-1-2-7-1701109335 with key local-on-demand-ip-1-1-2-7-1701109335
DEBU[0001] Found ETCDSnapshotFile for on-demand-ip-1-1-2-7-1701109315 with key s3-on-demand-ip-1-1-2-7-1701109315
DEBU[0001] Found ETCDSnapshotFile for on-demand-ip-1-1-2-7-1701109335 with key s3-on-demand-ip-1-1-2-7-1701109335
INFO[0001] Reconciliation of ETCDSnapshotFile resources complete