search

aliyun / aliyun-odps-jdbc

JDBC Driver for ODPS
Other
125 stars 38 forks source link

Proposal: spliting the source table when migrating data #9

Closed onesuper closed 8 years ago

onesuper commented 8 years ago

Apache Sqoop uses a trick to maximize the concurrency.

https://abgaik.wordpress.com/2013/06/09/splitting-data-in-apache-sqoop/

It relies on the distribution of numeric values.

We could use this trick to fully utilize the network bandwidth when uploading records to ODPS.

cc @lyman

onesuper commented 8 years ago

Done. The code is under /example