search

guzzijones / tdeBuilder

build Tableau tde from csv or odbc using a json configuration file
MIT License
3 stars 2 forks source link

row count limit #1

Open jibanes opened 7 years ago

jibanes commented 7 years ago

There seem to be a row count limitation, as shown below, is there a known workaround/fix?

python tdeBuilder.py -i extractDesc.json Totalrows: 614166 Build start time: 2017-01-26 10:50:17 Percent: 0.0 %, row: 1 Percent: 2.0 %, row: 12284 Percent: 4.0 %, row: 24567 Percent: 6.0 %, row: 36850 Percent: 8.0 %, row: 49134 Percent: 10.0 %, row: 61417 Percent: 12.0 %, row: 73700 Percent: 14.0 %, row: 85984 Percent: 16.0 %, row: 98267 Percent: 18.0 %, row: 110550 Percent: 20.0 %, row: 122834 Percent: 22.0 %, row: 135117 Percent: 24.0 %, row: 147400 Percent: 26.0 %, row: 159684 Percent: 28.0 %, row: 171967 tde file built to: test_table.tde End Time: 2017-01-26 10:51:15.753012 Build elapsed time: 0:00:58.627732 Uploaded to server: http://10.200.144.22

NOTE: This behavior is identical whereas the machine has 8GB of ram or 256GB (tested).

jibanes commented 7 years ago

I think this problem occurs when one of the field contains a "\n" character, as such the row count is the number of lines in the file, not the number of records, could you confirm this? In essence it's purely cosmetic.

guzzijones commented 7 years ago

I would need the input file. Could u post it on google docs?

guzzijones commented 7 years ago

I just saw your note about the cosmetic nature of the bug.

jibanes commented 7 years ago

yep, looks like just Totalraw's value is wrong with exotic CSVs, and, implicitely calculated %.

guzzijones commented 7 years ago

i am surprised this file even works and the error is only cosmetic. I certainly did not code for the edge case of having embedded line feeds inside quotes.

guzzijones commented 7 years ago

i have the file.