Tasks prefetching

[frol]

What is the recommended way of dealing with a heap of tasks?

It seems like there is a hard-coded 1-second delay somewhere (please, notice the logging timestamps between "task is finished" and "task is successful")...

2016-10-18 14:07:33,638 [INFO] [kuyruk.worker] Processing task: {'function': 'extract_dataset_info', 'sender_cmd': 'client.py', 'id': 'fe7f44be953b11e692320242ac110012', 'queue': 'kuyruk', 'sender_hostname': 'ab1b9d5de6b2', 'module': 'common', 'args': ['qq'], 'sender_pid': 110, 'sender_timestamp': '2016-10-18T14:05:58', 'kwargs': {}}
HEy!
2016-10-18 14:07:33,639 [INFO] [kuyruk.task] common:extract_dataset_info finished in 0 seconds.
2016-10-18 14:07:34,640 [INFO] [kuyruk.worker] Task is successful
2016-10-18 14:07:34,643 [INFO] [kuyruk.worker] Processing task: {'function': 'extract_dataset_info', 'sender_cmd': 'client.py', 'id': 'fe813d8a953b11e6ab700242ac110012', 'queue': 'kuyruk', 'sender_hostname': 'ab1b9d5de6b2', 'module': 'common', 'args': ['qq'], 'sender_pid': 110, 'sender_timestamp': '2016-10-18T14:05:58', 'kwargs': {}}
HEy!
2016-10-18 14:07:34,644 [INFO] [kuyruk.task] common:extract_dataset_info finished in 0 seconds.
2016-10-18 14:07:35,645 [INFO] [kuyruk.worker] Task is successful
2016-10-18 14:07:35,647 [INFO] [kuyruk.worker] Processing task: {'function': 'extract_dataset_info', 'sender_cmd': 'client.py', 'id': 'fe8382b0953b11e6aca90242ac110012', 'queue': 'kuyruk', 'sender_hostname': 'ab1b9d5de6b2', 'module': 'common', 'args': ['qq'], 'sender_pid': 110, 'sender_timestamp': '2016-10-18T14:05:58', 'kwargs': {}}
HEy!
2016-10-18 14:07:35,649 [INFO] [kuyruk.task] common:extract_dataset_info finished in 0 seconds.
2016-10-18 14:07:36,650 [INFO] [kuyruk.worker] Task is successful
2016-10-18 14:07:36,652 [INFO] [kuyruk.worker] Processing task: {'function': 'extract_dataset_info', 'sender_cmd': 'client.py', 'id': 'fe85b51a953b11e685560242ac110012', 'queue': 'kuyruk', 'sender_hostname': 'ab1b9d5de6b2', 'module': 'common', 'args': ['qq'], 'sender_pid': 110, 'sender_timestamp': '2016-10-18T14:05:58', 'kwargs': {}}
HEy!
2016-10-18 14:07:36,653 [INFO] [kuyruk.task] common:extract_dataset_info finished in 0 seconds.
2016-10-18 14:07:37,654 [INFO] [kuyruk.worker] Task is successful

[cenkalti]

This was a serious bug. Thank you for reporting 👍

[frol]

@cenkalti Wow! That was fast! Thank you!

[cenkalti]

I will release a new version to PyPI after fixing #41

[frol]

Just to confirm, I have tested the master and it works fast as expected now!

[cenkalti]

Enjoy! https://pypi.python.org/pypi/Kuyruk/4.1.2