Change default queue sizes in retry loops

[lucasmoten]

The retry loops can result in an overall flow deadlock if the queues between processing steps fill as depicted below

To counteract this, the size threshold should allow for something like

       number-of-inflows to the emitter * maximum file size being processed * retry counter

In practice, with the split routine, the max file size on a flow at the point of retry is 4GB And number-of-inflows to the emitter, is 2, with retry counter being 5.

This calls for raising the size of the queue to allow up to 40 GB.

[ghost]

It seems the only real way to programmatically change the default backpressure size threshold for a template is by modifying the nifi.properties file either through the rest API or physically replacing it before loading nifi. This will set the default for ALL connections, not just ones in template. I can’t find a way to create a custom connection and as it stands the connection properties don’t allow for expression language. We can set this default before nifi loads in our setup but this won’t necessarily help with changing default values of template connections if they are used outside of our personal nifi setup(i.e. another image is used, they have a custom properties file as well etc). We can also hardcode the size threshold property values in each template but that seems frivolous if we don’t know all use cases and would/will probably change to suit each individual case anyway.

[lucasmoten]

This was addressed in #34