search

dinedal / textql

Execute SQL against structured text like CSV or TSV
MIT License
9.06k stars 300 forks source link

Too many columns #74

Open Narven opened 6 years ago

Narven commented 6 years ago

On this website there is some csv files that I downloaded to test textql (the first one)

https://support.spatialkey.com/spatialkey-sample-csv-data/

when I run textql -sql "select count(*) from sample_data" FL_insurance_sample.csv I get

2018/04/09 13:57:58 too many columns on FL_insurance_sample

The file has huge amount of rows, but only contains 18 columns.

Is the error correct?

crazyvikas23 commented 6 years ago

The error is correct indeed. Above file contains CR line terminators(\r). You can verify it with file <filename>. Currently Golang csv parser used by textql do not support CR line terminators. You should replace all occurrences of \r with \n. See this Golang Issue.

Narven commented 6 years ago

Thanks for the feedback.

Can't the tool just detect the CR and fix the file accordingly?

In my opinion, people that could use the tool don't care about CR... for them is just a csv file, they just want for it to work.

If someone just sees that error... is going to immediately say... this tool is crap, which is not the case (in my opinion), just incomplete. Either the error needs to be much more explanatory or just make the tool fix the CR issue.

Also if someone is going to use this tool is because they have really huge csv, and nobody is going to open them to replace \r by \n.

Just an opinion.