Currently, on the app input side, the app puts the data into a queue which is then put into storage by the maintenance task. Likewise, on the CLA egress side, the maintenance task queues data to the CLA, which then pulls it out as it is ready to do so.
Both of these queues create challenges for data throttling, where it is not easy to provide backpressure to the queue-filling side when constraints occur on the queue-emptying side.
A possible alternative approach is to have the CLA directly pull data from the storage service, and app data directly push data to the storage service. This would necessitate different locking strategy because the service would be directly accessed by different tasks, but on the plus side, the approach would simplify things considerably, and solve some other concerns in the process, such as existing issues #102 and #108.
Currently, on the app input side, the app puts the data into a queue which is then put into storage by the maintenance task. Likewise, on the CLA egress side, the maintenance task queues data to the CLA, which then pulls it out as it is ready to do so.
Both of these queues create challenges for data throttling, where it is not easy to provide backpressure to the queue-filling side when constraints occur on the queue-emptying side.
A possible alternative approach is to have the CLA directly pull data from the storage service, and app data directly push data to the storage service. This would necessitate different locking strategy because the service would be directly accessed by different tasks, but on the plus side, the approach would simplify things considerably, and solve some other concerns in the process, such as existing issues #102 and #108.