search

keedio / flume-ng-sql-source

Flume Source to import data from SQL Databases
Apache License 2.0
264 stars 164 forks source link

Multi tables supports? #38

Open zhupishiye opened 7 years ago

zhupishiye commented 7 years ago

as there is a c3p0 ,one pool for one table ,sounds like a waste when need multi tables to sync ,rather than configuration multi flume sources?

mvalleavila commented 7 years ago

Hello,

You are right, using one pool for one connection is not the best solution. Now, the only way to configure more than one table is using more than one source.

The upgrade to support more than one table, can be develop a parser for multi table configuration in the source, something like:

[....] status.file.name.1 status.file.path.1 start.from.1 delimiter.entry.1 enclose.by.quotes.1 columns.to.select.1 run.query.delay.1 batch.size.1 max.rows.1 custom.query.1

status.file.name.2 status.file.path.2 start.from.2 delimiter.entry.2 enclose.by.quotes.2 columns.to.select.2 run.query.delay.2 batch.size.2 max.rows.2 custom.query.2 [....]

We will keep this issue open to remember the future upgrade. Anyway feel free to contribute with a pull request if you want.

Thank you for the proposal

Regards Marcelo

domenicovasile commented 6 years ago

Hello,

I'm using more than one source on the same agent to get records from a set of tables residing on the same database. It's quite annoying to replicate all db related configurations on each source, although this seems to be more a flume inherent issue. Furthemore hibernate still creates one c3p0 pool for each of the sources, despite there is only one jdbcUrl/user/password. And this is not a desirable behaviour at all. I'm not a hibernate expert but I suppose the problem could be the repeated instantiation of the SessionFactory (static) object before every call to openSession(). Can you please suggest a workaround? Thanks in advance