Set cgroups command to launch Flink stream applications

felipegutierrez / explore-flink

This project uses Apache Flink as a stream engine that consumes data from the File system or Kafka brokers and exposes metrics using Prometheus and Grafana, everything deployed on Kubernetes (minikube).

44 stars 22 forks source link

Set cgroups command to launch Flink stream applications #15

Closed felipegutierrez closed 3 years ago

felipegutierrez commented 4 years ago

We need to launch stream applications using cgroups in order to be able to set different quotas to each operator. It is necessary to add the command cgexec -g "cpulimited" <COMMAND to launch stream application> on the launch.sh script and test it.

abelpc commented 4 years ago

Is the following link somehow related? https://cwiki.apache.org/confluence/pages/viewpage.action?spm=a2c65.11461447.0.0.3bb3582dXndn6v&pageId=65147077

felipegutierrez commented 4 years ago

Somehow. For instance, when we deploy flink on Mesos and we have none application running we see none task managers available. Only when we deploy a stream application somehow the task managers appear. This is referred to as the dynamic resource allocation.

The farthest that Flink goes is defining Slots in each task manager. Then we can place a number of threads/operators in the slots Task chaining and resource groups.