search

gchq / Maestro

A framework for configurable operation executors
4 stars 5 forks source link

Create hooks to enable operation execution monitoring #2

Open d21211122 opened 5 years ago

d21211122 commented 5 years ago

It would be great if hooks were available that send out application monitoring messages that can be collected in a central monitoring system. Then information about operation chains that execute across a set of federated services can be gathered and visualised in one place.

So consider an operation chain ([operationA, executor1] -> [[operationB1 -> operationB2], executor2] -> [operationC, executor3]). The federated operation chain handler is going to run through these operations one by one, delegating to the relevant executor for each operation in the chain.

If each of the executors report back information about the operations they are carrying out and the context in which they are doing it (an executor id, the job id of the operation chain, the timestamp of beginning and end of operation, for example), then by aggregating this information we can see the state of the distributed workflow in one place.

In future, this information could be mined to inform users about their operations. It would be possible to estimate the runtime of an operation chain based on previous execution statistics, for example.

The logs could either be sent to an external logging system, or collected as part of the existing job tracker. We could pass the configured monitoring endpoint between systems using the operation context.

w122stern commented 5 years ago

I'll have a go at this please :)