Idea/Proposal: Tekton Data Pipeline Framework to Onboard Data

[alvin-reyes]

Overview

Once we have K8s installed on our EHI, we need to start looking into Data Onboarding Tools.

I propose the use of Tekton Data Pipeline (https://github.com/tektoncd/pipeline). This is essentially a task framework that leverages k8s service infrastructure to create ephemeral task runners in the form of pods/containers.

How it'll work.

*Queue is optional.

• We'll set up docker compose / dockerfile for ptolemy, delta and the downloader script i.e containerize them
• Set up a Tekton task for each.
• The downloader then assigns a batch to process for each ptolemy and delta.

[10d9e]

I love that this is k8s native - it should give us the capability to turn the appropriate dials to deal with scale. cc: @Zorlin

[alvin-reyes]

Adding here: I've been researching on the use of pyspark as the data processing task for this.