Spark / DeltaLake Support for KF

[rauerhans]

In order to work with the KF deployment, we require a data management solution. As discussed this should be a combination of Spark / DeltaLake on top of S3.

Requirements:

• Spark cluster is scalable / can be triggered to scale from the IDE / Notebook
• Spark jobs can triggered from wihin the notebook -> Authentication is given
• No YAMLS in the IDE to configure the setup / cluster (= interactive mode)
• PySpark Support
• Results are accessible via Notebook
• Solution has been tested with the official KF OpenGPT-X Deploy
• User Documentation is provided
• (Code can be executed via KF Pipeline - should be maybe yaml with sparkapplication)
• (spark config in background.)

[rauerhans]

Tim commented: spark delta lake works interactivly with 1 executor pod.

with 2 or more executor pods, a connection between each other it not possible yet…

IRSA testing is needed as the next step

[rauerhans]

Tim commented: IRSA testing/debugging/finding a solution was done yesterday

All infos can be found here: [https://discordapp.com/channels/880830238723047424/893111923778928640/941372492893790219|https://discordapp.com/channels/880830238723047424/893111923778928640/941372492893790219|smart-link]

Automation needed from plural side. Opened a github issue: [https://github.com/pluralsh/plural-artifacts/issues/159|https://github.com/pluralsh/plural-artifacts/issues/159|smart-link]