search

square / ETL

Extract, Transform, and Load data with Ruby
Other
386 stars 28 forks source link

Full vs. Incremental Loading #13

Open amigotune opened 8 years ago

amigotune commented 8 years ago

Hi!

It's not clear to me if Square does incremental or full loads. Is it able to do incremental ones? Where can I find more information about this?

Cheers!

jeffreyiacono commented 8 years ago

hey @amigoricardo happy to help answer your question, but wanted to make sure I understand it a bit more: are you asking if you can use this framework to do data processing that continues to build from a given starting point or if you need to do a complete rebuild each time? We've used this to do both and the amount of data and destination structure has really been the deciding factor in picking one set up over the other. Let me know!

amigotune commented 8 years ago

Hi! I want to sync two tables according to a specific query. Thus, I would like to Square to identify, at each run, the minimum amount of UPDATEs, INSERTs and DELETEs in order to sync them.