datalake/configs: rework iceberg enablement configs

bharathv commented 4 days ago

(new) redpanda.iceberg.delete = {true, false}, global configuration default iceberg_delete {true, false}
(update) redpanda.iceberg.mode, accepted values={disabled, key_value, value_schema_id_prefix} replaces redpanda.iceberg.enabled={true/false}
(delete) Removing iceberg_translation_interval_ms_default and redpanda.iceberg.translation.interval.ms

Additionally includes a fix for potentially translation gaps due to compaction.

Normally gaps shouldn't happen with translation due to the enforcement of max_collectible_offset. However the following sequence of actions can create a gap.

iceberg enabled iceberg disabled <-- compaction --> iceberg enabled

This is an unfixable gap if compaction cleaned up an offset range adjacent to last translated offset. The fix just plugs the gap by adjusting the begin offset of the range that is to be committed with the coordinator. This is a rare case but can result in a stuck translation if it happens.

Backports Required

[ ] none - not a bug fix
[ ] none - this is a backport
[ ] none - issue does not exist in previous branches
[ ] none - papercut/not impactful enough to backport
[x] v24.3.x
[ ] v24.2.x
[ ] v24.1.x

Release Notes

none

the below tests from https://buildkite.com/redpanda/redpanda/builds/58441#01934de2-cddb-4bf2-b3b3-1ad3b5f16b31 have failed and will be retried
```
translator_test_rpfixture
```
the below tests from https://buildkite.com/redpanda/redpanda/builds/58441#01934de2-cde0-4dfd-b064-deb1870f49a0 have failed and will be retried
```
translator_test_rpfixture
```
the below tests from https://buildkite.com/redpanda/redpanda/builds/58523#019351b0-b5b0-42d8-8117-a0cc541ab5c7 have failed and will be retried
```
translator_test_rpfixture
```
the below tests from https://buildkite.com/redpanda/redpanda/builds/58523#019351b0-b5b2-4caa-a0ca-a3a75071e11a have failed and will be retried
```
translator_test_rpfixture
```
the below tests from https://buildkite.com/redpanda/redpanda/builds/58574#01935491-afdb-402a-9172-6e4589ea745e have failed and will be retried
```
translator_test_rpfixture
```
the below tests from https://buildkite.com/redpanda/redpanda/builds/58574#01935491-afdc-4870-afb3-c1316bd4e06e have failed and will be retried
```
translator_test_rpfixture
```
the below tests from https://buildkite.com/redpanda/redpanda/builds/58594#01935547-f177-4465-8b68-23a51a1a7533 have failed and will be retried
```
translator_test_rpfixture
```
the below tests from https://buildkite.com/redpanda/redpanda/builds/58594#01935547-f17a-4add-8f26-e174b2ecdba2 have failed and will be retried
```
translator_test_rpfixture
```
the below tests from https://buildkite.com/redpanda/redpanda/builds/58614#019355fa-c6a1-4156-8d51-d71942deb329 have failed and will be retried
```
translator_test_rpfixture
```
the below tests from https://buildkite.com/redpanda/redpanda/builds/58614#019355fa-c6a4-47e4-acb3-1bcc02885c5c have failed and will be retried
```
translator_test_rpfixture
```
the below tests from https://buildkite.com/redpanda/redpanda/builds/58630#019356da-9d21-46d2-9322-75f7f918d1a1 have failed and will be retried
```
translator_test_rpfixture
```
the below tests from https://buildkite.com/redpanda/redpanda/builds/58630#01935721-7652-42bd-b7c6-0765ec6208ff have failed and will be retried
```
translator_test_rpfixture
```

bharathv commented 3 days ago

/dt

vbotbuildovich commented 3 days ago

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/58481#01935047-8947-485f-b31f-5cf421174d22:

"rptest.tests.datalake.rest_catalog_connection_test.RestCatalogConnectionTest.test_redpanda_connection_to_rest_catalog.cloud_storage_type=CloudStorageType.S3.use_serde_parquet=True"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/58481#01935047-8945-44ae-b536-dadf8713a3c5:

"rptest.tests.datalake.rest_catalog_connection_test.RestCatalogConnectionTest.test_redpanda_connection_to_rest_catalog.cloud_storage_type=CloudStorageType.S3.use_serde_parquet=False"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/58481#0193504a-8a2c-4b19-b2e1-5b8e7fc58e5c:

"rptest.tests.datalake.rest_catalog_connection_test.RestCatalogConnectionTest.test_redpanda_connection_to_rest_catalog.cloud_storage_type=CloudStorageType.S3.use_serde_parquet=True"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/58481#0193504a-8a38-4159-892b-1ca0aa2b864c:

"rptest.tests.datalake.rest_catalog_connection_test.RestCatalogConnectionTest.test_redpanda_connection_to_rest_catalog.cloud_storage_type=CloudStorageType.S3.use_serde_parquet=False"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/58523#0193520a-4f69-4593-946f-6551b73f4474:

"rptest.tests.datalake.coordinator_retention_test.CoordinatorRetentionTest.test_retention.cloud_storage_type=CloudStorageType.S3"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/58523#019351f7-0f3c-4f02-9cce-e4183c526459:

"rptest.tests.datalake.coordinator_retention_test.CoordinatorRetentionTest.test_retention.cloud_storage_type=CloudStorageType.S3"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/58574#019354eb-5466-4a19-9021-bca8ae9f8e74:

"rptest.tests.datalake.coordinator_retention_test.CoordinatorRetentionTest.test_retention.cloud_storage_type=CloudStorageType.S3"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/58594#019355a1-4bf1-4961-8219-5c104eb4c759:

"rptest.tests.polaris_catalog_smoke_test.PolarisCatalogSmokeTest.test_connecting_to_catalog.cloud_storage_type=CloudStorageType.S3.with_tls=True"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/58594#019355a1-4bf6-41e6-9905-2aaa9caf93ce:

"rptest.tests.polaris_catalog_smoke_test.PolarisCatalogSmokeTest.test_connecting_to_catalog.cloud_storage_type=CloudStorageType.S3.with_tls=False"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/58614#01935651-afe5-4198-a64e-6c0eb7a09dbf:

"rptest.tests.polaris_catalog_smoke_test.PolarisCatalogSmokeTest.test_connecting_to_catalog.cloud_storage_type=CloudStorageType.S3.with_tls=False"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/58614#01935651-afde-4afe-a063-993d2eda99b3:

"rptest.tests.polaris_catalog_smoke_test.PolarisCatalogSmokeTest.test_connecting_to_catalog.cloud_storage_type=CloudStorageType.S3.with_tls=True"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/58614#01935656-ff17-402f-abe2-15b67488e8d0:

"rptest.tests.polaris_catalog_smoke_test.PolarisCatalogSmokeTest.test_connecting_to_catalog.cloud_storage_type=CloudStorageType.S3.with_tls=False"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/58614#01935656-ff1a-4173-a45e-e2123a263309:

"rptest.tests.polaris_catalog_smoke_test.PolarisCatalogSmokeTest.test_connecting_to_catalog.cloud_storage_type=CloudStorageType.S3.with_tls=True"

vbotbuildovich commented 3 days ago

Retry command for Build#58481

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/datalake/rest_catalog_connection_test.py::RestCatalogConnectionTest.test_redpanda_connection_to_rest_catalog@{"cloud_storage_type":1,"use_serde_parquet":true}
tests/rptest/tests/datalake/rest_catalog_connection_test.py::RestCatalogConnectionTest.test_redpanda_connection_to_rest_catalog@{"cloud_storage_type":1,"use_serde_parquet":false}

vbotbuildovich commented 3 days ago

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/58481#01935047-8943-4f97-8a01-8d23d8300b36 ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/58481#0193504a-8a2c-4b19-b2e1-5b8e7fc58e5c ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/58523#019351f7-0f3d-487a-952d-abffbe431a71 ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/58523#0193520a-4f6d-4b79-a245-87922ff91240 ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/58523#0193520a-4f6b-4f8c-bab8-87df84c25535 ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/58523#019351f7-0f38-4dd9-9639-3f39bd242cde

vbotbuildovich commented 2 days ago

Retry command for Build#58523

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/datalake/coordinator_retention_test.py::CoordinatorRetentionTest.test_retention@{"cloud_storage_type":1}
tests/rptest/tests/datalake/datalake_e2e_test.py::DatalakeE2ETests.test_e2e_basic@{"cloud_storage_type":1,"filesystem_catalog_mode":true,"query_engine":"spark"}
tests/rptest/tests/datalake/datalake_e2e_test.py::DatalakeE2ETests.test_e2e_basic@{"cloud_storage_type":1,"filesystem_catalog_mode":true,"query_engine":"trino"}

bharathv commented 2 days ago

A new fix landed in dev that conflicts with this PR, rebasing shortly.

vbotbuildovich commented 2 days ago

Retry command for Build#58574

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/datalake/coordinator_retention_test.py::CoordinatorRetentionTest.test_retention@{"cloud_storage_type":1}

vbotbuildovich commented 2 days ago

Retry command for Build#58594

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/polaris_catalog_smoke_test.py::PolarisCatalogSmokeTest.test_connecting_to_catalog@{"cloud_storage_type":1,"with_tls":true}
tests/rptest/tests/polaris_catalog_smoke_test.py::PolarisCatalogSmokeTest.test_connecting_to_catalog@{"cloud_storage_type":1,"with_tls":false}

bharathv commented 2 days ago

Retry command for Build#58594

please wait until all jobs are finished before running the slash command
/ci-repeat 1
tests/rptest/tests/polaris_catalog_smoke_test.py::PolarisCatalogSmokeTest.test_connecting_to_catalog@{"cloud_storage_type":1,"with_tls":true}

Another new test added during rebase that was referring to a removed config 💀

vbotbuildovich commented 2 days ago

Retry command for Build#58614

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/polaris_catalog_smoke_test.py::PolarisCatalogSmokeTest.test_connecting_to_catalog@{"cloud_storage_type":1,"with_tls":false}
tests/rptest/tests/polaris_catalog_smoke_test.py::PolarisCatalogSmokeTest.test_connecting_to_catalog@{"cloud_storage_type":1,"with_tls":true}

vbotbuildovich commented 1 day ago

/backport v24.3.x

vbotbuildovich commented 1 day ago

Failed to create a backport PR to v24.3.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-24222-v24.3.x-765 remotes/upstream/v24.3.x
git cherry-pick -x aa17e5e6c6 5cbd42abf8 882af4db57 e0fabee560 54c42df9d1 8eca184766 8f0fc0494f 23829f562d 1ba9882e8d a81ee528ef a87e6ce9c7 2d923519b6

Workflow run logs.

redpanda-data / redpanda