search

keedio / flume-ng-sql-source

Flume Source to import data from SQL Databases
Apache License 2.0
264 stars 164 forks source link

imports a few rows from mysql and stops #30

Closed simran-k closed 6 years ago

simran-k commented 7 years ago

I am using hive for sink and 8 rows got imported in table and then it stopped. It is not streaming at all. My sink is hive.

I have a couple of more problems:

I see two problems here:

  1. NULLS, There should be data in some columns, instead it keeps showing nulls. This used to happen in CSV's as well but then I had set CSV serde and it worked. How does it work here with ORC. This data is being streamed through flume using flume-ng-sql from mysql database.
  2. How do I get rid off double quotes here?
  3. Why are the rows breaking at comma. Although at the time of table creation, I specified ROW FORMAT DELIMITED "\003", how does a comma break it then?
mvalleavila commented 7 years ago

Hello @simran-k ,

Sorry for the delay in the response. Are you still having the problem importing the rows?

Answering your questions:

  1. I will try to reproduce it, and solve in a new release
  2. I'm not sure about it, I will try to reproduce this behavior too and see how the source handle the double quotes.
  3. With the last pull request by lufeng1102 it's solved.

Regards