tarantool / test-run

Tarantool functional testing framework
14 stars 14 forks source link

dispatcher: lift pipe buffer size restriction #434

Closed Totktonada closed 1 month ago

Totktonada commented 1 month ago

A task queue dispatcher puts all the tasks to the task queue at startup. Then workers are started and are taking the tasks from it.

If there are many tasks in a task group (which roughly corresponds to a test suite), we can reach the pipe buffer size on putting into the queue, because multiprocessing.SimpleQueue uses a pipe under the hood.

The solution is to use multiprocessing.Queue, which has an intermediate buffer before the underlying pipe and writes to the pipe in a background thread, without blocking a thread that calls <queue>.put().

The Queue API is a superset of the SimpleQueue API, so we can just replace the implementation.

Let's also use Queue for the worker's output queue to be on the safe side and for consistency.

Fixes #287

coveralls commented 1 month ago

Coverage Status

coverage: 62.54% (+0.03%) from 62.506% when pulling 584c1283af7f9028ae5dfa93b85e650a7acfe866 on Totktonada/gh-287-lift-pipe-buffer-size-restriction into 10372993105ad3cb6162dc90061eda9d5243c40f on master.