lmco / streamflow

StreamFlow™ is a stream processing tool designed to help build and monitor processing workflows.
https://github.com/lmco/streamflow/wiki
Apache License 2.0
253 stars 69 forks source link

Persistent topology monitor to ensure uptime of submitted topologies. #40

Open juliencruz opened 9 years ago

juliencruz commented 9 years ago

Currently, when a topology is submitted by a user from streamflow there are no protections that it will continue to run if there is a cluster outage.

This feature should implement a persistent service within the streamflow server that ensures that a topology that has been submitted will be resubmitted if the topology goes down for any reason. This will improve topology uptime in the event that a storm cluster goes down and comes back up for any reason.

Once a user triggers the kill operation in streamflow, the topology will be flagged as killed and should no longer be auto deployed.

The streamflow configuration should allow for enabling/disabling of this feature and controlling the polling interval of the service. A suggested config format is as follows:

monitor:
    enabled: true
    pollingInterval: 60