XML-RPC multithreading seems broken

[cidermole]

When trying to feed Moses via XML-RPC with 32 sentences to translate in parallel, it returns occasional duplicates. Depending on the method to do the threading in Python, sometimes Moses blocks on and never replies to XML-RPC requests.

Standalone Python3 script to feed XML-RPC and trigger this bug: http://pastebin.com/fJdGQywq

Moses config (pretty straightforward, CBPT enabled [1]). http://pastebin.com/rmtLdxFg

Script results (32 lines, but note the duplicates): http://pastebin.com/1E42ngD7

Expected script results (no duplicates): http://pastebin.com/gRZ1QuGa

(obtained without threading, by waiting for each request as follows)

for l in emea_en[0:32]:
    t = threading.Thread(target=translate_thread, args=(l,))
    t.start()
    t.join()

[1] http://www.statmt.org/moses/?n=Advanced.CacheBased

[wangpd]

Hi,

I am thinking whether you made a typo in your python client which caused the problem. In the following function, I guess you want to use "text" (the argument to

the function) instead of "l" on Line 3:

def translate_thread(text): proxy_n = xmlrpclib.ServerProxy(moses_url) response = proxy_n.translate({'text': l})

print(response['text'])

, as the "l" is from the main thread and it could be any arbitrary sentence.

Hope it helps.

Best wishes! Pidong

On 30 July 2015 at 12:17, cidermole notifications@github.com wrote:

When trying to feed Moses via XML-RPC with 32 sentences to translate in parallel, it returns occasional duplicates. Depending on the method to do the threading in Python, sometimes Moses blocks on and never replies to XML-RPC requests.

Standalone Python3 script to feed XML-RPC and trigger this bug: http://pastebin.com/fJdGQywq

Moses config (pretty straightforward, CBPT enabled [1]). http://pastebin.com/rmtLdxFg

Script results (32 lines, but note the duplicates): http://pastebin.com/1E42ngD7

Expected script results (no duplicates): http://pastebin.com/gRZ1QuGa

(obtained without threading, by waiting for each request as follows)

for l in emea_en[0:32]: t = threading.Thread(target=translate_thread, args=(l,)) t.start() t.join()

[1] http://www.statmt.org/moses/?n=Advanced.CacheBased

— Reply to this email directly or view it on GitHub https://github.com/moses-smt/mosesdecoder/issues/120.

[cidermole]

Ouch. This one was really obvious, thank you.

I'm still wondering why the other attempt of mine is blocking indefinitely...

But indeed, this looks like it has nothing to do with Moses.

[wangpd]

You are welcome.

Best wishes! Pidong

On 30 July 2015 at 14:32, cidermole notifications@github.com wrote:

Ouch. This one was really obvious, thank you.

I'm still wondering why the other attempt of mine is blocking indefinitely...

But indeed, this looks like it has nothing to do with Moses.

— Reply to this email directly or view it on GitHub https://github.com/moses-smt/mosesdecoder/issues/120#issuecomment-126496486 .