job hangs on creating a thread

[hu6360567]

I wrote a Task Class with 2 subprcoess objects running client/server. It logs a status every 1 second while the subprocesses are running.

class e2eTask(Task):
    def run(self, params):
        port = params['port']
        volume = params['volume']  # pure digits, unit MB
        # utility = params['utility']

        task = TransferTask(port=port, volume=volume + 'M')
        task.runServer()

        startTime = time.time()
        task.runClient()
        log.info("Before isRunning")
        while task.server.proc.poll():
            log.info("Run %.1f seconds, transferring %s" % (time.time() - startTime, task.transferred()))
            time.sleep(1)
        task.wait()

It works fine without sending it into the queue.

If I send it into a queue, it hangs at task.runClient() until it exists. The client runs the client subprocess and a thread to capture subprocess output.

    def run(self):

        def enqueue_output(out, queue):
            for line in iter(out.readline, b''):
                queue.put(line)
            out.close()

        self.proc = Popen(self.command, stdout=PIPE, bufsize=1)
        self.queue = Queue()
        self.thread = Thread(target=enqueue_output, args=(self.proc.stdout, self.queue))
        self.thread.daemon = True
        self.thread.start()

How should I write task to solve this problem?

[hu6360567]

Using gevent patch solves.

from gevent.monkey import patch_all
patch_all(thread=False)

but has a Runtime Warning:

Patching more than once will result in the union of all True parameters being patched

Is there a better solution?

[sylvinus]

Hi @hu6360567.

Indeed, because MRQ uses gevent out of the box, you have to include that line to make the thread module behave correctly. You can safely ignore the warning.

Please note that we don't recommend using threads with MRQ, if you have a CPU-intensive task you should use multiple processes and IO-intensive tasks should use greenlets instead.