ke4roh / vertx-engine

An execution engine for processing things, based on vertx
Apache License 2.0
1 stars 2 forks source link

Collect an array of values #38

Open ke4roh opened 5 years ago

ke4roh commented 5 years ago

register drops a value into a spot. Great. But what if we need to assemble an array of indefinite length? A practical example is collecting return values from a list of some N searches where the number N is not known a priori. Each value should be atomically appended to an array.

LightGuard commented 5 years ago

I'm not sure why we need to do anything different here. The execute of a step returns a Maybe<JsonObject> If you need to build something up in the execute, simply add it to the JsonObject and return when you're done.

If you want multiple things to be able to register to the same location we could support that, but not sure what the need would be.

ke4roh commented 5 years ago

Possibly. My main concern was a race condition with asynchronous tasks completing near-simultaneously. If two steps were to fire their events to update the doc before either one actually disposes of the event, the result of the first one is lost. (Perhaps this is impossible due to the vert.x threading model - I'm not certain yet.)

LightGuard commented 5 years ago

I think it might be best to leave it as is. Be a nice citizen and don't register to the same location. We could do a sanity check of a pipeline when we load it and spit out any warnings like this.

ke4roh commented 5 years ago

I want to collect results from a list of searches. There may be M searches, of which only N execute depending on their applicability to the input. I don't want the aggregator to need to know about each of the M searches, just that it should aggregate from a list.

A flag on abstract step invocation can fire the event with a special header to append, and the engine can process append events differently.

LightGuard commented 5 years ago

Depends on #39