datalake/tests: enable parametrizing datalake tests across query engines / catalog modes

[bharathv]

Refactors the interfaces to abstract out query engine specifics into a base class and enables plugging in required SQL engines at test time. Refactored existing end-to-end with parameterization as an example.

Backports Required

• [ ] none - not a bug fix
• [ ] none - this is a backport
• [x] none - issue does not exist in previous branches
• [ ] none - papercut/not impactful enough to backport
• [ ] v24.2.x
• [ ] v24.1.x
• [ ] v23.3.x

Release Notes

• none

  the below tests from https://buildkite.com/redpanda/redpanda/builds/57569#0192fa2c-acc7-4b61-823f-69af6ff6857a have failed and will be retried

  datalake_cloud_rpunit

  the below tests from https://buildkite.com/redpanda/redpanda/builds/57569#0192fa2c-acc8-4ebb-9712-785179d17afd have failed and will be retried

  catalog_schema_manager_rpunit

  the below tests from https://buildkite.com/redpanda/redpanda/builds/57599#0192fcce-131f-46a7-8429-281da8d8b0c3 have failed and will be retried

  catalog_schema_manager_rpunit

[vbotbuildovich]

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/57599#0192fd0f-bc17-4086-b5df-486a906a823f:

"rptest.tests.control_character_flag_test.ControlCharacterPermittedAfterUpgrade.test_upgrade_from_pre_v23_2.initial_version=.22.2.9"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/57599#0192fd2b-fbd3-4ca5-af00-04fb5f654ff2:

"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_learner_gap_metrics"

[vbotbuildovich]

Retry command for Build#57599

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/control_character_flag_test.py::ControlCharacterPermittedAfterUpgrade.test_upgrade_from_pre_v23_2@{"initial_version":[22,2,9]}
tests/rptest/tests/nodes_decommissioning_test.py::NodesDecommissioningTest.test_learner_gap_metrics

[bharathv]

failure unrelated, haven't touched any non datalake test code.

requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://vectorized-public.s3.us-west-2.amazonaws.com/releases/redpanda/22.3.25/redpanda-22.3.25-amd64.tar.gz