queue.submit() iterates over chunks, sends each, and then resets the chunks. This can be a problem if any process adds items to the queue while submit() is iterating. When that happens, the new additions can be cleared out before they can be sent.
An alternative is to repeatedly pop the first chunk while the chunks aren't empty. I'll open a pull request.
queue.submit() iterates over chunks, sends each, and then resets the chunks. This can be a problem if any process adds items to the queue while
submit()is iterating. When that happens, the new additions can be cleared out before they can be sent.An alternative is to repeatedly
popthe first chunk while thechunksaren't empty. I'll open a pull request.