Enable filter to use ipyparalell

HDFGroup / datacontainer

Data Container Study

Other

8 stars 1 forks source link

Enable filter to use ipyparalell #7

Closed jreadey closed 8 years ago

jreadey commented 8 years ago

Add option to filter script (e.g. summary.py) to enable use of ipyparallel.

jreadey commented 8 years ago

I've converted summary.py to work with ipyparallel.

Runs fine on my home linux box with 8 engines, but am seeing heartbeat failures with ODSC.

jreadey commented 8 years ago

The heartbeat failures were due to the engine nodes running out of memory trying to do too many simultaneous s3cmd downloads. I've updated the code to limit downloads to 5 processes at a time and that has resolved the issue.