Currently when a new route opens up (such as for a CLA coming up) the storage service will scan all previously-received idle bundles (those for which it is not been able to forward yet) and immediately move them into the outgoing queue for processing.
This works when only a small number of bundles are waiting, but does not scale well. Assuming there could be thousands of matching bundles in storage and the uplink rate and duration might only accommodate a few of them, it would be wasteful to reevaluate every matching bundle.
There should be a source throttling mechanism where the scanning service will only locate route-matching bundles up to a limit. This is somewhat implemented by the depth limit on the storage output queue, but if this queue gets filled then it will block ALL routes out of the storage, not just this one. Therefore that is not quite sufficient, the scanning service needs a way to synchronize its rate with the CLA, rather than filling the intermediate queue(s) to the limit.
Previous PR #101 laid some of the groundwork to allow this to happen, but needs more work.
Currently when a new route opens up (such as for a CLA coming up) the storage service will scan all previously-received idle bundles (those for which it is not been able to forward yet) and immediately move them into the outgoing queue for processing.
This works when only a small number of bundles are waiting, but does not scale well. Assuming there could be thousands of matching bundles in storage and the uplink rate and duration might only accommodate a few of them, it would be wasteful to reevaluate every matching bundle.
There should be a source throttling mechanism where the scanning service will only locate route-matching bundles up to a limit. This is somewhat implemented by the depth limit on the storage output queue, but if this queue gets filled then it will block ALL routes out of the storage, not just this one. Therefore that is not quite sufficient, the scanning service needs a way to synchronize its rate with the CLA, rather than filling the intermediate queue(s) to the limit.
Previous PR #101 laid some of the groundwork to allow this to happen, but needs more work.