QUESTION: limiting concurrency of a node *TYPE*

[aarav191]

Say we have a server that implements each request as a simple flow graph: A->B->C Each request creates a new graph. Q: if B is memory intensive, how can we limit the number of B nodes executing at the same time across all graph instances? All of the examples in the documentation seem to apply concurrency limits to a node within a single graph and not across graphs.

[aleksei-fedotov]

The only way is to count the number of B nodes and their concurrency limitation across all instantiated graphs and make a decision on the concurrency limitation of the new B node including delaying of a new graph instantiation if the limit is reached.

However, it is quite unusual to me that instantiation of a new graph is necessary each time the request comes. I would ask why not just push the message into the existing graph hence starting the processing pipeline once the request comes? Why new instantiation is needed?

Consider the attached image showing how limiting of in-flight message number can be achieved using limiter_node. I have added the decider multifunction_node, which the only purpose is to decide onto what message processor to send a new message. This is needed if you have a B node with different processing logic each time the new message comes. For this graph, you can specify unlimited concurrency for the B nodes, and, of course, make the application running in the background with an instantiated graph awaiting a new message to be passed by the server process:

Hope this helps.

[aarav191]

Thanks Aleksei, this is very helpful!

why not just push the message into the existing graph hence starting the processing pipeline once the request comes?

Yeah, I started thinking along those lines. One difficulty with that is that the documentation always specifies to "Always Use wait_for_all()" and in general that causes difficulties with a shared flow graph. Is it strictly necessary to call wait_for_all() for processing to continue, or is it sufficient to just call it before shutdown?

As an aside, would the limiter node approach you outlined work across unrelated flow graphs to limit maximum memory usage (at least among memory hogging nodes), or is there a better strategy for that?

[aleksei-fedotov]

Is it strictly necessary to call wait_for_all() for processing to continue, or is it sufficient to just call it before shutdown?

Well, wait_for_all() is strictly necessary to call in order to guarantee the processing. Without it the graph processing may just not happen. For example, consider the case of a single-threaded application - the main thread is the only thread in the application, that thread pushes messages to the graph and that very thread processes them once it calls graph::wait_for_all(), because there is no any other thread to do that. This is what being said by the phrase "oneTBB does not guarantee any parallelism". So, answering your question, we may now say that it is sufficient to call wait_for_all() before the shutdown.

work across unrelated flow graphs

The problem with edges between graphs is that it can easily become cumbersome to figure out in what order to call the wait_for_all() on these graphs to guarantee that all messages are processed. Please see this section in oneTBB developer guide regarding edges between graphs.

In general, I recommend reading the whole Flow Graph Tips and Tricks section, and, particular to your case - the section of Limiting Resource Consumption.

Hope this helps!

[isaevil]

@aarav191 is this issue still relevant or should we close it?