search

apache / incubator-heron

Apache Heron (Incubating) is a realtime, distributed, fault-tolerant stream processing engine from Twitter
https://heron.apache.org/
Apache License 2.0
3.64k stars 598 forks source link

Update k8's Scheduler Code #3259

Open joshfischer1108 opened 5 years ago

joshfischer1108 commented 5 years ago

The scheduler code is out of date and needs to be updated to use the InCluster API in the Java Kubernetes client. By updating the code we can remove the proxy container that is being used today in the Helm chart.

This following class should be deprecated and use the AppsV1 APIs going forward.

https://github.com/apache/incubator-heron/blob/master/heron/schedulers/src/java/org/apache/heron/scheduler/kubernetes/AppsV1beta1Controller.java#L52

the context class will have to change we no longer need a scheduler URI when running inside the cluster

https://github.com/apache/incubator-heron/blob/master/heron/schedulers/src/java/org/apache/heron/scheduler/kubernetes/KubernetesContext.java#L28

An example of getting the cluster config is below:

https://github.com/apache/pulsar/blob/master/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/KubernetesRuntimeFactory.java#L254

nicknezis commented 4 years ago

I'm working on this. Have changes needed for the 6.0.1 version of the Java API. Have not yet made changes to use the in cluster config, but I'll make that change also.

nicknezis commented 4 years ago

3426 and #3432 should complete this issue. Might still be worth updating the scheduler code to remove the scheduler URI context config item.

joshfischer1108 commented 4 years ago

@nicknezis

3426 and #3432 should complete this issue. Might still be worth updating the scheduler code to remove the scheduler URI context config item.

@nicknezis Could you make an issue and describe what would need to be done to remove the scheduler URI context config item?