Issue with Pathos and Amazon EC2'cluster

[rtadewald]

I'm having some trouble trying to run some map calculations using a cluster i developed in Amazon EC2 with a software called StarCluster.

Actually, i have a master instance and a node and they can ssh to each other easily. The problem is that i can't make they work together. I'm using the following code:

rhost = '54.147.215.185'
port = 22
t = SSH_Tunnel('Tunnel')
lport = t.connect(rhost, port)

ppservers = []
ppservers.append("localhost:" + str(lport))
ppservers = tuple(ppservers)
job_server = pp.Server(ppservers=ppservers)

After that, i got the following message: Known servers: [('local',)] [('localhost', 50631)]

But, when i try to do some calculation, all the calcuation is done on the master node.

Job execution statistics: job count | % of all jobs | job time sum | time per job | job server 8 | 100.00 | 6.3714 | 0.796429 | local

What am i doing wrong?

[mmckerns]

Sorry this slipped through the cracks somehow. Is this still an issue?

The default is to split jobs from the Pool between the local and other ppservers. So, I think what you need to do is use set_ncpus to set the number of local ppservers.

If you add:

job_server.set_ncpus(0)

it should only remote ppservers

Let me know if this issue can be closed..

[mmckerns]

I'm assuming that if this doesn't answer your question, you will reopen the ticket.