search

treasure-data / luigi-td

Luigi Workflow Engine integration for Treasure Data
http://docs.treasuredata.com/articles/luigi
Apache License 2.0
16 stars 15 forks source link

Add new option parallel and prepare parallel #6

Closed yudppp closed 7 years ago

yudppp commented 7 years ago

Hi @k24d , @toru-takahashi , @yyuu

Add bulk import option parallel and prepare parallel.

--prepare-parallel specifies the number of threads are to be used to compress the data locally. Normally, this number should match the number of CPU cores on your machine. (https://docs.treasuredata.com/articles/bulk-import-tips-and-tricks)

I think you should prepare-parallel option default value is CPU cores, but this occurr small breaking changes.

Because so prepare-parallel option default is as it is.

coveralls commented 7 years ago

Coverage Status

Coverage increased (+0.01%) to 94.928% when pulling a64e64e7549977e78f882fb56a399ea9aa7affd5 on yudppp:feature/add_parallel_and_prepare-parallel_options into e06edc0d9ffb9652746a8b45cd4067c273cf3701 on treasure-data:master.

k24d commented 7 years ago

@yudppp thank you. This change looks good and I'll merge it.

k24d commented 7 years ago

Oops, I can't :( @yyuu can I ask you to take over?

yudppp commented 7 years ago

Thanks:)

yyuu commented 7 years ago

Released v0.6.8 on PyPI including this.

https://pypi.python.org/pypi/luigi-td/0.6.8