search

harelba / q

q - Run SQL directly on delimited files and multi-file sqlite databases
http://harelba.github.io/q/
GNU General Public License v3.0
10.19k stars 421 forks source link

Can't change encoding #226

Closed asgerhallas closed 3 years ago

asgerhallas commented 4 years ago

Hi,

On windows, q v2.0.9, no matter which encoding I use except for "utf-8", I get this message:

AttributeError("'UnicodeDecodeError' object has no attribute 'message'")

or for some encodings...

AttributeError("'UnicodeError' object has no attribute 'message'")

For example when I do this:

q -tH -e "ascii" "SELECT * FROM data.tsv WHERE titleType = 'movie'"

What am I doing wrong? :)

harelba commented 4 years ago

hi, sorry for the too-late reply, crazy times

could you please provide some more information about your system, and perhaps an example file (just the first few lines, or a synthetic file which recreates the issue) so i can investigate?

christophgysin commented 3 years ago

Here's an example:

$ echo -e "\xe6" > test.csv
$ q "SELECT * FROM test.csv"
AttributeError("'UnicodeDecodeError' object has no attribute 'message'")
asgerhallas commented 3 years ago

(@harelba sorry for my late reply too. I missed yours it seems, but @christophgysin's message got me in here again)

harelba commented 3 years ago

This character is on the upper half of ASCII, and the standard ascii encoding in python doesn't support that other half.

Use the iso-8859-1 encoding, it's kind of an extended ASCII which supports the upper half of the characters.

$ q -e iso-8859-1 "SELECT * FROM test.csv"
Warning: column count is one - did you provide the correct delimiter?
æ
$