Closed JCZuurmond closed 1 year ago
We need this feature and curious about that
@armsepehr tell me more about what you need here? There are several different ways to approach this and I haven't hit upon one that I'm happy with yet, so this context would be really helpful for me in figuring it out.
(FWIW, I'm racing to get issue #147 done asap, so I think the write-side plugins will have to wait for a post-1.5.0 release)
We read information from the file and then do the transformation pipeline on data. Finally, we would like to store transformed data in another db such as Clickhouse, AlloyDB or other relational dbs. I've watched out #141 and think about the external destination to write data.
okay, that makes sense and fits with how I was thinking about this stuff-- thanks! It's going to involve some major surgery tho, so I'm not going to get to it until I get the 1.5.0 stuff out the door; sorry about that!
The same as in #141, though, now for writing data to external destinations.
Motivation
The reasoning is similar to reading. Sometimes the data has to be presented in a different system. For example, I use
jdbc
tables in Spark to write data to a SQL server. The SQL server might be connected to a dashboard tool, which can be preferred over connecting the Spark cluster.The general approach for the plugins is fantastic! Users could implement anything; maybe they want to send an email with a report summarizing the data quality of a data model.