search

Middlecon / DBImport

DBImport ingestion tool. Handle import, export and standard ETL flows in Hadoop/Hive
Apache License 2.0
16 stars 6 forks source link

Message about 'split-by-text' even if the column is an integer #5

Closed Middlecon closed 5 years ago

Middlecon commented 5 years ago

Starting an import with a custom query and trying to use more than one mapper, you get a message saying that you are not allowed to split by a text field. You get this even if you are spliting by an integer column

Middlecon commented 5 years ago

The sqoop-query we generate includes " around the column name .This works for the query, but the build in function in sqoop to get min/max values will identify this as a text value and the output column is a char column.

We solved this by generating a boundaryQuery based on the split-by column and specifying the columnname without " in that query. This also opens up for adding custom queries for min/max in the future.