search

bartongroup / slivka

http://bartongroup.github.io/slivka/
Apache License 2.0
7 stars 3 forks source link

TaskQueue communication, sending incomplete messages. #55

Closed warownia1 closed 5 years ago

warownia1 commented 5 years ago

When the job is submitted to the QueueServer it initially reads 16 bytes to determine the header and the content length. The full request content, or even the header, may not be complete the moment the socket becomes "readable". This would raise BlockingIOError as not enough data is read from the socket. Possible solutions are using blocking sockets (maybe with some asynchronous library like gevent or asyncio) or buffering the message until it's fully complete (16 bytes header + bytes was received) and processing the request afterwards.

As for now, there was no problems. If any issues occur, changing conn.send(head + content) to conn.sendall(head + content) in QueueServer.communite should be the first thing to try.

warownia1 commented 5 years ago

Using redis to transfer data between the scheduler and the queue is another option.