edanalytics / lightbeam

CLI tool for validating and transmitting payloads from JSONL files into an Ed-Fi API.
Apache License 2.0
10 stars 1 forks source link

Better handling of leftover tasks in a (potentially large) task queue #41

Open tomreitz opened 3 months ago

tomreitz commented 3 months ago

This thread includes discussion of how to restructure the processing of task queues in batches so that we don't have to include a final block to process any leftover tasks in the queue. @johncmerfeld proposed a way to do that but @tomreitz pointed out possible memory impact of his solution - opening this issue so we don't forget to consider better solutions.