Managing (submitting, killing, etc) Topology at Run-time programmatically

[merttgokalp]

I was able to deploy topologies programatically with Storm using this code-snippet; System.setProperty("storm.jar", "/.../heron-examples.jar"); StormSubmitter.submitTopology("myTopology", conf, builder.createTopology());

However, Heron do not allow deploying topologies at run-time. I got the error; Exception in thread "main" java.lang.RuntimeException: topology definition temp directory not specified

Are there any solution to deploy topologies programmatically/at run-time with Heron ?

[kramasamy]

@hellowBigData - Can you give an example program of how you submit it?

[merttgokalp]

@kramasamy This code snippet works for Storm to deploy topologies.

TopologyBuilder builder = new TopologyBuilder();
    builder.setSpout("word", new TestWordSpout(), 1);
    builder.setBolt("exclaim1", new ExclamationBolt(), 1)
        .shuffleGrouping("word");
    Config conf = new Config();
    config.setMaxTaskParallelism(3);
    config.setMessageTimeoutSecs(10);
    config.setNumWorkers(2);
    System.setProperty("storm.jar", "/Users/merttgokalp/Desktop/heron-examples.jar");
    StormSubmitter.submitTopology("myTopology", conf, builder.createTopology());

The problem is that HeronSubmitter.java file only reads the command line parameters to deploy a topology. Therefore, I can not deploy a topology programmatically at run-time using Java.

  if (heronCmdOptions.get("cmdline.topologydefn.tmpdirectory") != null) {
      submitTopologyToFile(fTopology, heronCmdOptions);
    } else {
      throw new RuntimeException("topology definition temp directory not specified");
    }

[kramasamy]

@hellowBigData - thanks for the snippet. We will look into this. In our use cases, we do not submit programmatically. Hence, we did not implement this functionality.

[avitorovic]

A related comment: It would also be nice to have a way to programmatically kill a topology. In Storm, I could do it with backtype.storm.generated.Nimbus.Client.killTopology(topologyName). This is useful when data sources send some kind of EOF signal, after which we might decide to kill the topology.

[kramasamy]

@avitorovic - thanks for the feedback.

[leonardgithub]

I also get the "topology definition temp directory not specified" error programmatically, any solution?