search

coursera / dataduct

DataPipeline for humans.
Other
252 stars 83 forks source link

Support for workergroups #209

Closed skullmaggots closed 8 years ago

skullmaggots commented 8 years ago

Hi,

In my pipelines I would like to use workergroups as I have already created VMs which run the TaskAgent process. It doesn't look like this is currently supported but is a mandatory feature IMHO.

Thanks,

Jeff.

sb2nov commented 8 years ago

@skullmaggots worker groups are already supported. @cliu587 can you point to the correct documentation for this ?

AaronTorgerson commented 8 years ago

@skullmaggots It's not documented, but I found the configuration for this:

In your dataduct.cfg:

ec2:
    WORKER_GROUP: MyWorkerGroup
...

That's it. You don't need anything else in your ec2 config block. Hope that helps (though, I realize you asked this almost 2 months ago).

cliu587 commented 8 years ago

I updated the documentation (found here: http://dataduct.readthedocs.org/en/latest/config.html#ec2). Thank you for this =)