Closed gohalo closed 1 month ago
Duplicate files were returned from the remote storage. The most likely cause is an S3 or HDFS API compatibility issue with your remote storage implementation. duplicate file:
Caused by the incompatible API to S3 api by huawei OBS parallel file system.
ROOT CAUSE:
When server side processing the v2 listing request as v1 listing request, and in case of need pagination, the pagination mark is different from v1 and v2, which causes the client side missing interpreting the pagination mark and continuing the list from beginning again, and never ending. In order to break this endless loop, a duplication check is added to detect the problem and crash the BE/CN loudly because manual interfere is required for such case.
TEMP REMEDIATION
ROOT CAUSE:
- AWS CPP SDK is used to access all S3-compatible object storage. It is the listObjectsV2 API by default to list objects on remote service.
- OBS parallel filesystem implements listObjectsV1 API, not the listObjectsV2 API and not reporting error when the client issues a listObjectV2 API to an listObjectsV1 endpoint.
When server side processing the v2 listing request as v1 listing request, and in case of need pagination, the pagination mark is different from v1 and v2, which causes the client side missing interpreting the pagination mark and continuing the list from beginning again, and never ending. In order to break this endless loop, a duplication check is added to detect the problem and crash the BE/CN loudly because manual interfere is required for such case.
TEMP REMEDIATION
- Find the error path from the log
- Drop the entire table for partition forcibly
- manual clean the data to the path on obs console
- start BE/CN again
so that means the number of files under some obs dirs, should not larger than pagination size, maybe 1000?
yes
partially fixed in https://github.com/StarRocks/starrocks/pull/48949
branch-3.1 related fix in https://github.com/StarRocks/starrocks/pull/49103
3.1.13 share-data mode