can patch connectorx to petl to speed in next version?

petl-developers / petl

Python Extract Transform and Load Tables of Data

MIT License

1.25k stars 193 forks source link

can patch connectorx to petl to speed in next version? #619

Open wonb168 opened 2 years ago

wonb168 commented 2 years ago

my 700,0000 rows table , use petl to csv need 40 miniutes. and use connectorx, parrelled by 10, only 15miniutes, can patch connectorx to petl to speed in next version?

juarezr commented 2 years ago

It looks like a very promising feature:

petl could benefit from connector-x query parallelism for speed.
connector-x could benefit from petl functionality for on-the-fly transformation, cleansing, manipulation, calculations

References:

wonb168 commented 2 years ago

petl's fromdb just one sql to query source db, but connector-x can split the sql to many, for exmaple: select * from tb, id, 4, if 100 rows in table , then will 4 sql to query table, select * from tb where id<=25 I mean , if petl's fromdb funtion use connector-x to speed.