R2 does not work with S3 endpoint | aws: s3 write file: failed to create multipart upload: NotImplemented

[8times4]

Describe the bug Dragonfly throws an error when trying to save to an R2 bucket.

To Reproduce Steps to reproduce the behavior:

Quick docker reproduction command: (still requires a cloudflare r2 account)

 docker run --rm -p 6379:6379 -e AWS_ACCESS_KEY_ID=<r2 access key> -e AWS_SECRET_ACCESS_KEY=<r2 secret key> -e AWS_REGION=us-east-1 --ulimit memlock=-1 docker.dragonflydb.io/dragonflydb/dragonfly:latest --dir s3://dflydb-prod --logtostderr --requirepass=secretpass --s3_endpoint=<accountid>.r2.cloudflarestorage.com --s3_use_https=true

Try to either just cmd/ctrl+c the container, or try to send a SAVE command to it from a redis-cli

Expected behavior Expected to save to R2.

Logs

I20240528 13:10:03.215538     1 init.cc:70] dragonfly running in opt mode.
I20240528 13:10:03.215699     1 dfly_main.cc:646] Starting dragonfly df-v1.18.1-6851a4c845625b0b14bb145177322dafbbc9858e
W20240528 13:10:03.215804     1 dfly_main.cc:685] SWAP is enabled. Consider disabling it when running Dragonfly.
I20240528 13:10:03.215816     1 dfly_main.cc:690] maxmemory has not been specified. Deciding myself....
I20240528 13:10:03.215817     1 dfly_main.cc:699] Found 4.32GiB available memory. Setting maxmemory to 3.46GiB
W20240528 13:10:03.215864     1 dfly_main.cc:373] Weird error 1 switching to epoll
I20240528 13:10:03.217207     1 proactor_pool.cc:147] Running 8 io threads
I20240528 13:10:03.222258     1 server_family.cc:721] Host OS: Linux 6.7.12-orbstack-00201-g2ddb8f197a46 aarch64 with 8 threads
I20240528 13:10:03.235483    12 snapshot_storage.cc:185] Creating AWS S3 client; region=us-east-1; https=true; endpoint=<account_id>.r2.cloudflarestorage.com
I20240528 13:10:03.235543    12 credentials_provider_chain.cc:28] aws: disabled EC2 metadata
I20240528 13:10:03.236860    12 credentials_provider_chain.cc:36] aws: loaded credentials; provider=environment
I20240528 13:10:03.240021    13 snapshot_storage.cc:242] Load snapshot: Searching for snapshot in S3 path: s3://dflydb-prod/
W20240528 13:10:03.485853     1 server_family.cc:814] Load snapshot: No snapshot found
I20240528 13:10:03.497128     9 listener_interface.cc:101] sock[19] AcceptServer - listening on port 6379
^CI20240528 13:10:04.933329     8 accept_server.cc:25] Exiting on signal Interrupt
I20240528 13:10:04.935065     9 listener_interface.cc:219] Listener stopped for port 6379
E20240528 13:10:05.040671     9 s3_write_file.cc:137] aws: s3 write file: failed to create multipart upload: NotImplemented
E20240528 13:10:05.125510    15 s3_write_file.cc:137] aws: s3 write file: failed to create multipart upload: NotImplemented
E20240528 13:10:05.130617     9 s3_write_file.cc:137] aws: s3 write file: failed to create multipart upload: NotImplemented
E20240528 13:10:05.132378    11 s3_write_file.cc:137] aws: s3 write file: failed to create multipart upload: NotImplemented
E20240528 13:10:05.139791    14 s3_write_file.cc:137] aws: s3 write file: failed to create multipart upload: NotImplemented
E20240528 13:10:05.144387     8 s3_write_file.cc:137] aws: s3 write file: failed to create multipart upload: NotImplemented
E20240528 13:10:06.125360    13 s3_write_file.cc:137] aws: s3 write file: failed to create multipart upload: NotImplemented
E20240528 13:10:06.140058    10 s3_write_file.cc:137] aws: s3 write file: failed to create multipart upload: NotImplemented
E20240528 13:10:06.144856    12 s3_write_file.cc:137] aws: s3 write file: failed to create multipart upload: NotImplemented
W20240528 13:10:06.148021     8 server_family.cc:841] Failed to perform snapshot Input/output error: Failed to open write file

Environment (please complete the following information):

• OS: [ubuntu 22.04 / MacOs 14.5]
• Containerized?: [k8s via operator & Docker]
• Dragonfly Version: [latest]

Additional context R2's S3 api support: https://developers.cloudflare.com/r2/api/s3/api/

Thank you!

• 8x4

[SoGooDFR]

Same with GCS

E20240806 12:55:00.687247     8 s3_write_file.cc:96] aws: s3 write file: failed to complete multipart upload: MalformedCompleteMultipartUploadRequest
E20240806 12:55:00.709810     9 s3_write_file.cc:96] aws: s3 write file: failed to complete multipart upload: MalformedCompleteMultipartUploadRequest
E20240806 12:55:00.720940    12 s3_write_file.cc:96] aws: s3 write file: failed to complete multipart upload: MalformedCompleteMultipartUploadRequest
E20240806 12:55:00.724735    11 s3_write_file.cc:96] aws: s3 write file: failed to complete multipart upload: MalformedCompleteMultipartUploadRequest
E20240806 12:55:00.739326    13 s3_write_file.cc:96] aws: s3 write file: failed to complete multipart upload: MalformedCompleteMultipartUploadRequest
E20240806 12:55:00.742241    14 s3_write_file.cc:96] aws: s3 write file: failed to complete multipart upload: MalformedCompleteMultipartUploadRequest
E20240806 12:55:00.754492    10 s3_write_file.cc:96] aws: s3 write file: failed to complete multipart upload: MalformedCompleteMultipartUploadRequest
E20240806 12:55:00.771561    15 s3_write_file.cc:96] aws: s3 write file: failed to complete multipart upload: MalformedCompleteMultipartUploadRequest
E20240806 12:55:00.877135    15 s3_write_file.cc:96] aws: s3 write file: failed to complete multipart upload: MalformedCompleteMultipartUploadRequest
W20240806 12:55:00.877169    15 server_family.cc:1034] Failed to perform snapshot Input/output error```