search

Cascading / cascading-jdbc

cascading schemes and taps for JDBC
http://cascading.org
27 stars 15 forks source link

Performance scaling: key column not taken to decide input spilit #25

Open poruspurush opened 9 years ago

poruspurush commented 9 years ago

Hi,

Original issue posted here, https://groups.google.com/forum/#!topic/lingual-user/ixaDrCxT_DI

We are exploring JDBC tap to pull the database from RDBMS. we are finding that basic select statements built by cascading JDBC API is not utilizing indexed columns (i.e. primary columns).

Is there anything we are missing that could influence the JDBC tap to utilize the PK index?

Thanks, Purush

fs111 commented 9 years ago

The problem you are facing has to do with the InputFormat used by cascading jdbc. It seems that the sqoop folks have a more clever implementation called DataDrivenInputFormat. Something like that would be very useful in cascading-jdbc as well.

That being said for your concrete case I would propose you create a sub-class of OracleDBInputFormat that matches your data better: https://github.com/Cascading/cascading-jdbc/blob/wip-2.7/cascading-jdbc-oracle/src/main/java/cascading/jdbc/db/OracleDBInputFormat.java and use that isntead of the one that we ship.

If you can come up with an easy way to roll your changes into a portalbe/generic code, I am more than happy to integrate that into the project.