search

skinkie / reference

Personal repository where I collect working examples to understand inner workings while building PyNeTExConv
GNU Affero General Public License v3.0
0 stars 1 forks source link

fix output encoding #12

Closed ue71603 closed 2 months ago

skinkie commented 2 months ago

It should be done differently. Some function where Python implements 'iconv' functionality hence, open in encoding 1 write it to UTF-8. Then DuckDB can naively load it as UTF-8.

ue71603 commented 2 months ago

My tools don't complain about the input files. They are utf-8. I think somehting within python went wrong. Or open does not use utf-8 by default.

ue71603 commented 2 months ago

I did the trick from https://stackoverflow.com/questions/36303919/what-encoding-does-open-use-by-default

print(locale.getpreferredencoding())

on my machine it returned: cp1252

So that's what open was using. I guess you don't hve that on your unix machine.

This will not end before python 3.15: https://peps.python.org/pep-0686/

ue71603 commented 2 months ago

meaning: my way is ok :-)

skinkie commented 2 months ago

meaning: my way is ok :-)

Only then we keep it.