storage: The S3-compatible API of Google Cloud Storage does not support DeleteObjects

kennytm commented 7 months ago

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Create a GCS bucket.
Create a TiCDC v7.6 changefeed with redo-log enabled (setting consistent.level = 'eventual'), and set the storage to s3://bucket/?endpoint=http://storage.googleapis.com (thus accessing GCS through their S3-compatible XML API).
Wait for a while.

2. What did you expect to see? (Required)

No error.

3. What did you see instead (Required)

Changefeed stopped because of incompatible API:

[2024/02/01 00:00:00.000 +00:00] [WARN] [meta_manager.go:159] ["redo: init redo meta fail"] [namespace=default] [changefeed=«redacted»] [error="[CDC:ErrS3StorageAPI]external storage api: NotImplemented: A header or query you provided requested a function that is not implemented.\n\tstatus code: 400, request id: , host id: "]

4. What is your TiDB version? (Required)

v7.6.0

kennytm commented 7 months ago

No this is not an issue about TiCDC. TiCDC uses the external storage API in the BR package, thus the labels.
The error happened because TiCDC since pingcap/tiflow#10097 used the DeleteFiles() function introduced from #47084.
On S3 (and KS3), this maps to the DeleteObjects (POST /?delete) API for batch deletion in a single call. (Meanwhile other storage implementations just loop and delete each file one-by-one, including GCS).
The DeleteObjects API is not supported in the GCS XML API. This caused the "NotImplemented" error shown in the issue description.

The GCS JSON API (gcs://) uses an entirely different approach for batch delete (https://cloud.google.com/storage/docs/batch), so I believe there is no plan for Google to support DeleteObjects in the XML API.

For this issue we either

declare using s3:// for GCS is bad, users should just use gcs://, and close this issue, or
do support s3:// for GCS, make DeleteFiles() detect the HTTP 400, and fallback to use a loop

BornChanger commented 6 months ago

@kennytm can you please open the issue to tiflow repo instead? cc @flowbehappy

BornChanger commented 6 months ago

/remove-component br

kennytm commented 6 months ago

@BornChanger I have explained why the issue is filed here in the first line in https://github.com/pingcap/tidb/issues/50918#issuecomment-1922586687. I'm keeping the labels unless there is a dedicated label for the external-storage component.

pingcap / tidb