search

apache / incubator-heron

Apache Heron (Incubating) is a realtime, distributed, fault-tolerant stream processing engine from Twitter
https://heron.apache.org/
Apache License 2.0
3.64k stars 598 forks source link

Install aurora scheduler locally #1722

Open objmagic opened 7 years ago

objmagic commented 7 years ago

I'm surprised heron-core target does not include Aurora scheduler. Not really sure why.

./
./release.yaml
./heron-core/
./heron-core/bin/
./heron-core/bin/heron-executor
./heron-core/bin/heron-shell
./heron-core/bin/heron-stmgr
./heron-core/bin/heron-tmaster
./heron-core/bin/heron-python-instance
./heron-core/lib/
./heron-core/lib/scheduler/
./heron-core/lib/scheduler/heron-scheduler.jar
./heron-core/lib/scheduler/heron-local-scheduler.jar
./heron-core/lib/scheduler/heron-slurm-scheduler.jar
./heron-core/lib/scheduler/heron-mesos-scheduler.jar
./heron-core/lib/scheduler/heron-marathon-scheduler.jar
./heron-core/lib/packing/
./heron-core/lib/packing/heron-roundrobin-packing.jar
./heron-core/lib/packing/heron-binpacking-packing.jar
./heron-core/lib/metricsmgr/
./heron-core/lib/metricsmgr/heron-metricsmgr.jar
./heron-core/lib/statemgr/
./heron-core/lib/statemgr/heron-localfs-statemgr.jar
./heron-core/lib/statemgr/heron-zookeeper-statemgr.jar
./heron-core/lib/instance/
./heron-core/lib/instance/heron-instance.jar
objmagic commented 7 years ago

any idea why? @billonahill @kramasamy

billonahill commented 7 years ago

No idea. I wonder why heron-core includes any schedulers. I'd expect them to be distributed separately to reduce bloat and extra unnecessary deps.

maosongfu commented 7 years ago

One thing to notice is that: scheduler will also be run in server side, for instance, yarn-scheduler in container-0. If we exclude scheduler jars from heron-core, then we need to take care of downloading necessary scheduler jars properly.

But I agree at least heron-core should not contain all different implementations of scheduler.