Create a KNative Sample App that can create arbitrary spark streaming jobs

StreamState / terraform-k8s-configuration

1 stars 1 forks source link

Create a KNative Sample App that can create arbitrary spark streaming jobs #2

Open danielhstahl opened 3 years ago

danielhstahl commented 3 years ago

Explore how an app can use kubectl apply -f to create arbitrary spark streaming jobs.

Acceptance criteria:

Dockerfile with app
App must accept REST Api
App must call kubectl
App must take SQL through the REST Api and create a spark streaming job from it

I had originally thought this would run on the same k8s cluster, but now I think this could be a separate autoscaling instance...no need to have this be tied directly to the k8s instance per client.

danielhstahl commented 3 years ago

Thinking out loud...this may be better as a github workflow. Provisioning spark streaming jobs and running kubectl on the cluster. Out of the box monitoring/alerting (via github actions), and better security.

danielhstahl commented 3 years ago

https://strimzi.io/blog/2019/10/25/kafka-authentication-using-oauth-2.0/