Open fulmicoton opened 1 month ago
I did an experiment where I created stale splits and then started quickwit where delete requests to the S3 client would fail with 90% probability.
It did successfully retry until the splits were deleted and then returned in a idle state.
2024-07-23T08:08:00.462Z INFO quickwit_janitor::actors::garbage_collector: loaded 3 indexes from the metastore
2024-07-23T08:08:00.462Z INFO quickwit_storage::object_storage::s3_compatible_storage: using S3 credentials defined in storage config
2024-07-23T08:08:00.462Z INFO quickwit_storage::object_storage::s3_compatible_storage: using S3 region defined in storage config region=us-west
2024-07-23T08:08:00.462Z INFO quickwit_common: using environment variable `QW_S3_FORCE_PATH_STYLE_ACCESS` default value value=true
2024-07-23T08:08:00.462Z INFO quickwit_storage::object_storage::s3_compatible_storage: using S3 endpoint defined in storage config or environment variable endpoint=http://127.0.0.1:9000
2024-07-23T08:08:00.462Z INFO quickwit_storage::object_storage::s3_compatible_storage: using S3 credentials defined in storage config
2024-07-23T08:08:00.462Z INFO quickwit_storage::object_storage::s3_compatible_storage: using S3 region defined in storage config region=us-west
2024-07-23T08:08:00.462Z INFO quickwit_common: using environment variable `QW_S3_FORCE_PATH_STYLE_ACCESS` default value value=true
2024-07-23T08:08:00.462Z INFO quickwit_storage::object_storage::s3_compatible_storage: using S3 endpoint defined in storage config or environment variable endpoint=http://127.0.0.1:9000
2024-07-23T08:08:00.462Z INFO quickwit_storage::object_storage::s3_compatible_storage: using S3 credentials defined in storage config
2024-07-23T08:08:00.462Z INFO quickwit_storage::object_storage::s3_compatible_storage: using S3 region defined in storage config region=us-west
2024-07-23T08:08:00.462Z INFO quickwit_common: using environment variable `QW_S3_FORCE_PATH_STYLE_ACCESS` default value value=true
2024-07-23T08:08:00.462Z INFO quickwit_storage::object_storage::s3_compatible_storage: using S3 endpoint defined in storage config or environment variable endpoint=http://127.0.0.1:9000
[quickwit-storage/src/object_storage/s3_compatible_storage.rs:804:16] rand::random::<f64>() = 0.41916115486052985
[quickwit-storage/src/object_storage/s3_compatible_storage.rs:804:46] val = 0.9
2024-07-23T08:08:00.464Z WARN delete_splits_marked_for_deletion{index_uid=IndexUid { index_id: "simian_chico", incarnation_id: Ulid(2081418938362966106885799749050468484) } updated_before_timestamp=1721722020}: quickwit_sto
rage::object_storage::s3_compatible_storage: Bulk delete FAKE ERR
2024-07-23T08:08:00.464Z ERROR delete_splits_marked_for_deletion{index_uid=IndexUid { index_id: "simian_chico", incarnation_id: Ulid(2081418938362966106885799749050468484) } updated_before_timestamp=1721722020}: quickwit_ind
ex_management::garbage_collection: Failed to delete split file(s) ["01J3F9GHTMMDJE28ZMQYNHJ3YK.split", "01J3F9FFENDSQ4MHVA6702PKDE.split", "01J3F93GRHZEJ26D5RPXFYFR2P.split", "01J3F8ZC8PS82YTE2NBQW7V752.split", "01J3F97TS9PY
E4ZYT0M7V524CV.split", and 205 more] from storage. error=None index_id="simian_chico"
2024-07-23T08:08:10.468Z INFO quickwit_janitor::actors::garbage_collector: loaded 3 indexes from the metastore
2024-07-23T08:08:10.468Z INFO quickwit_storage::object_storage::s3_compatible_storage: using S3 credentials defined in storage config
2024-07-23T08:08:10.468Z INFO quickwit_storage::object_storage::s3_compatible_storage: using S3 region defined in storage config region=us-west
2024-07-23T08:08:10.468Z INFO quickwit_common: using environment variable `QW_S3_FORCE_PATH_STYLE_ACCESS` default value value=true
2024-07-23T08:08:10.468Z INFO quickwit_storage::object_storage::s3_compatible_storage: using S3 endpoint defined in storage config or environment variable endpoint=http://127.0.0.1:9000
2024-07-23T08:08:10.469Z INFO quickwit_storage::object_storage::s3_compatible_storage: using S3 credentials defined in storage config
2024-07-23T08:08:10.469Z INFO quickwit_storage::object_storage::s3_compatible_storage: using S3 region defined in storage config region=us-west
2024-07-23T08:08:10.469Z INFO quickwit_common: using environment variable `QW_S3_FORCE_PATH_STYLE_ACCESS` default value value=true
2024-07-23T08:08:10.469Z INFO quickwit_storage::object_storage::s3_compatible_storage: using S3 endpoint defined in storage config or environment variable endpoint=http://127.0.0.1:9000
2024-07-23T08:08:10.469Z INFO quickwit_storage::object_storage::s3_compatible_storage: using S3 credentials defined in storage config
2024-07-23T08:08:10.469Z INFO quickwit_storage::object_storage::s3_compatible_storage: using S3 region defined in storage config region=us-west
2024-07-23T08:08:10.469Z INFO quickwit_common: using environment variable `QW_S3_FORCE_PATH_STYLE_ACCESS` default value value=true
2024-07-23T08:08:10.469Z INFO quickwit_storage::object_storage::s3_compatible_storage: using S3 endpoint defined in storage config or environment variable endpoint=http://127.0.0.1:9000
[quickwit-storage/src/object_storage/s3_compatible_storage.rs:804:16] rand::random::<f64>() = 0.9919412888981723
[quickwit-storage/src/object_storage/s3_compatible_storage.rs:804:46] val = 0.9
2024-07-23T08:08:10.471Z WARN delete_splits_marked_for_deletion{index_uid=IndexUid { index_id: "simian_chico", incarnation_id: Ulid(2081418938362966106885799749050468484) } updated_before_timestamp=1721722030}: quickwit_sto
rage::object_storage::s3_compatible_storage: Bulk delete PROCEED
2024-07-23T08:08:11.663Z INFO delete_splits_marked_for_deletion{index_uid=IndexUid { index_id: "simian_chico", incarnation_id: Ulid(2081418938362966106885799749050468484) } updated_before_timestamp=1721722030}: quickwit_met
astore::metastore::file_backed::file_backed_index: deleted 0 splits from index index_id=simian_chico
2024-07-23T08:08:11.674Z INFO quickwit_janitor::actors::garbage_collector: Janitor deleted {"01J3F96QRSV6PZTJY2Z1TXY5SB.split", "01J3F99B5HVHVX97H23KQJJEJQ.split", "01J3F96455T16RZKB7KW8QM2ND.split", "01J3F97N6MVAHEWDKJFM81
JE4T.split", "01J3F9127GHCMAD734T1GMNABF.split"} and 210 other splits. index_id=simian_chico num_deleted_splits=210
@PSeitz Can you discuss with @esatterwhite to get details about the observed issue? Do you have access to airmail logs?
Once you get more info about the problem, can you edit the description of this ticket. Ping me if you get blocked on the fix.
Info from eric was that it stopped logging for several hours and he tries to reproduce. I don't have access to the logs
as reported on airmail.