search

keedio / flume-ng-sql-source

Flume Source to import data from SQL Databases
Apache License 2.0
264 stars 163 forks source link

"enclose.by.quotes" not functioning #53

Open Hypnotec opened 6 years ago

Hypnotec commented 6 years ago

E> It appears that 'enclose.by.quotes' does not function when using 'custom.query'

E> I might advocate that anyone who comes across this project consider switching to StreamSets, as Flume doesn't support RDBMS and StreamSets provides support for a large array of sources and targets without relying on third-party support.

Does the following property, "enclose.by.quotes", not function when using PSQL? I am not able to get it to work while using the following Flume properties:

#
# SOURCE
agent.sources.source-sql.type = org.keedio.flume.source.SQLSource
agent.sources.source-sql.channels = channel-file

agent.sources.source-sql.status.file.path = /home/admin/.tmp/.flume
agent.sources.source-sql.status.file.name = source-sql.messages.status

agent.sources.source-sql.hibernate.connection.url = jdbc:postgresql://localhost:5432/test
agent.sources.source-sql.hibernate.connection.user = postgres
agent.sources.source-sql.hibernate.connection.password = ''
agent.sources.source-sql.hibernate.connection.driver_class = org.postgresql.Driver

# NOTE: 'message_time' is integer, 'message' is text
agent.sources.source-sql.custom.query = select message_time, message from messages where message_time > $@$ order by message_time
agent.sources.source-sql.enclose.by.quotes = false

agent.sources.source-sql.run.query.delay = 5000
agent.sources.source-sql.batch.size = 500
agent.sources.source-sql.max.rows = 10000

Currently, my work around is to remove all instances of quotes through an interceptor, but I am not a huge fan of this approach as users may input double-quotes - which the 'message' field would capture.

lazaromedina commented 6 years ago

Hi Hypnotec, when trying to reproduce like issue #44, we get values with double quotes.

best, Luis