search

apache / incubator-heron

Apache Heron (Incubating) is a realtime, distributed, fault-tolerant stream processing engine from Twitter
https://heron.apache.org/
Apache License 2.0
3.65k stars 597 forks source link

how to reduce cpu resource requirement? #3494

Open dttlgotv opened 4 years ago

dttlgotv commented 4 years ago

After I write a very simple example , a error is hint when I submit it like below:

org.apache.heron.scheduler.SubmitterMain: Exception when submitting topology org.apache.heron.spi.packing.PackingException: Invalid packing plan generated. No enough CPU to allocate for unspecified instances at org.apache.heron.packing.roundrobin.RoundRobinPacking.calculateInstancesResourceMapInContainer(RoundRobinPacking.java:355) at org.apache.heron.packing.roundrobin.RoundRobinPacking.packInternal(RoundRobinPacking.java:173) at org.apache.heron.packing.roundrobin.RoundRobinPacking.pack(RoundRobinPacking.java:143) at org.apache.heron.scheduler.utils.LauncherUtils.createPackingPlan(LauncherUtils.java:71) at org.apache.heron.scheduler.SubmitterMain.submitTopology(SubmitterMain.java:444) at org.apache.heron.scheduler.SubmitterMain.main(SubmitterMain.java:334)

[2020-03-21 11:51:43 +0800] [ERROR]: Invalid packing plan generated. No enough CPU to allocate for unspecified instances [2020-03-21 11:51:43 +0800] [ERROR]: Failed to launch topology 'Test2Topology'

This topology is : public static void main(String[] args) throws Exception { Builder builder = Builder.newBuilder();

/**
 * The processing graph consists of a supplier streamlet that emits
 * random integers between 1 and 100. From there, a series of transformers
 * is applied. At the end of the graph, the original value is ultimately
 * unchanged.
 */
builder.newSource(() -> ThreadLocalRandom.current().nextInt(100))
      .log();

Config config =  Config.newBuilder()
    .setNumContainers(1)
    .setPerContainerRam(512)
    .setPerContainerCpu(1)
    .build();

// Fetches the topology name from the first command-line argument
String topologyName = StreamletUtils.getTopologyName(args);

// Finally, the processing graph and configuration are passed to the Runner, which converts
// the graph into a Heron topology that can be run in a Heron cluster.
new Runner().run(topologyName, config, builder);

} }

nwangtw commented 4 years ago

The default cpu config is 1 for each component. Container requires some overhead hence there is no way to schedule instances into containers. To fix the problem, you need to config the cpu requirement for each component to be less than 1.

I have created a PR to show how to config. https://github.com/apache/incubator-heron/pull/3496 For now we have to use the low level config because the it is not convenient to config components with the Streamlet config class.