Add interactive ETL demo

streamshub / flink-sql-examples

Apache License 2.0

1 stars 6 forks source link

Add interactive ETL demo #20

Closed tomncooper closed 1 week ago

tomncooper commented 1 week ago

This PR adds a simple tutorial for an ETL process using Flink SQL:

It is based on a Flink Session cluster running in minikube and port-forwarding the job manager's REST API port and running the sql-client from the flink-sql-runner image in docker locally.
It goes over some basic data cleaning and loading the cleaned data back into Kafka.
I would like to have loaded the data back using the Avro format rather than csv, but there didn't seem to be an easy way to do that from within Flink SQL. There may be a way to it externally via the Apicurio API which I will look into and add in a later PR.

tinaselenge commented 1 week ago

Thanks for the PR. Nicely written demo! Regarding the setup.sh script, I see a similar script in @SamBarker's PR #17. Should we perhaps come up with a common script for setting up everything, and starts up a different demo app based on the given argument or something?

tomncooper commented 1 week ago

@tinaselenge Good point about @SamBarker's script. I should have checked the open PRs! A common way to set-up the data generator would be a good thing. We will be adding more demos soon and it would be good not to re-invent the wheel.

@SamBarker is there much work needed to get your script merged. I am happy to help if you don't have time?

SamBarker commented 1 week ago

@SamBarker is there much work needed to get your script merged. I am happy to help if you don't have time?

Oh f*ck sorry. I think it needs the "upstream" branch fixed or removed (too add back in a separate PR). Feel free to push to that branch if you have the cycles

tomncooper commented 1 week ago

Thanks for the reviews folks. @SamBarker I am gonna merge this PR so we have the tutorial up. But we should def work to combine the data gen setup scripts and have a common setup for all demos.