search

portals-project / portals

Portals is a framework for stateful serverless apps, unifying dataflow streaming with actors
https://www.portals-project.org/
Apache License 2.0
19 stars 2 forks source link

Flink and Kafka as a compilation target #106

Open jspenger opened 1 year ago

jspenger commented 1 year ago

Compiling Portals applications to Flink and Kafka would be great for several reasons. First, it would act as a baseline, and we could have something to directly compare the performance against for our benchmarks. Second, It would be great for educational purposes, to very directly show what structure is necessary to support the programming model with existing Flink and Kafka runtimes. Lastly, it would not be too much effort, due to the existing Portal rewriting and the similarity between the programming models.

The core task is to implement a code generation step in the compiler that can generate and output a combination of Flink and Kafka programs. Additionally, a runtime which can execute these new programs through executing Flink and Kafka programs respectively, should be created. The runtime should also have some metadata management. The goal would be to be able to run an autoscaling deployment of Flink and Kafka on Kubernetes, and have a replicated Portals job server, which can accept new jobs and schedule them accordingly.

jspenger commented 1 year ago

On second thought, it would be just as fine to have Kafka + Kafka Streams as the compilation target.