search

square / ETL

Extract, Transform, and Load data with Ruby
Other
385 stars 28 forks source link

Is it possible to load data from Source to Dest between servers? #11

Open shut000 opened 9 years ago

shut000 commented 9 years ago

Is it possible to load data from Source to Dest between servers? Like what ETL usually does. Thanks! And I don't know where to add questions, just put it here...

jeffreyiacono commented 9 years ago

hmm, you'd probably need to expand the ETLs class's ability to hold more than one connection (source_connection / destination_connection) so you could pull from one server and send to another. Let me know if that makes sense and answers your question.

shut000 commented 9 years ago

thank you! I tried another one (activewarehouse-etl), it's more complex and support multiple servers. I'm new in Ruby and trying to break it down...

Emerson commented 9 years ago

interesyed in this functionality as well. Wondering what the API might look like.

jeffreyiacono commented 8 years ago

@Emerson @shut000 was just giving this a look over and have an idea on what could work here: right now we have ETL.connection that's used to speak with a single db via #query. This works fine when doing data processing within the same server / db, but totally see your use case. My thought: add a ETL.read_connection and ETL.write_connectionthat you can set and then interact with through two new methods#readand#writewhich would use either{read,write}_connection#query`. One requirement is we'd need to hold the read data in memory and then prepare it to be written, but that shouldn't been too bad!

Happy to take a pass at doing the above, let me know if it's still something you're interested in!