lunarmodules / luasql

LuaSQL is a simple interface from Lua to a DBMS.
http://lunarmodules.github.io/luasql
539 stars 191 forks source link

Seemly there is no way to set the encoding settings #65

Open speeder opened 7 years ago

speeder commented 7 years ago

I have a program here using LuaSQL that keeps getting mojibake out of the server, I am in Brazil, and it return brazillian-related stuff wrong.

raw byte dumps of the DB, and of the luasql results, show it is converting utf8 in the DB to something that looks like ucs2

for example, my first first name is Hélder, the é character in utf8 is c3a9, and this is what the DB dumps, and also the return value when using "SELECT HEX", but the byte dump when querying with luasql is E9, that is the ucs2 encoding of the character.

I did a "SHOW VARIABLES" on LuaSQL, and it defaults to latin1 and swedish, but I couldn't figure how to configure it to connect properly in first place.

The best I could find is that I would need the function "mysql_options" but I didn't found it in the source code here anywhere. (I am using binary distribution, so maybe I still missed it).

hishamhm commented 7 years ago

What do the $LANG and $LC_ALL variables look like in your environment? (Both in the client and server machines.) This doc for MySQL 5.7 indicates that using something like pt_BR.UTF-8 would make MySQL use UTF-8, but apparently MySQL 4.0 defaulted to Swedish so maybe you're running an old version.

speeder commented 7 years ago

The server started in an old MySQL, some tables are in latin1 Swedish because of this, but currently it defaults to UTF-8 (it is MySQL 5.7 too). All variables it prints if I use the MySQL client or PHP-MyAdmin are UTF-8

But the client with LuaSQL defaults to latin1 and swedish.

I asked MySQL people about this, and they said that the MySQL C library do default to this behaviour, and that the correct usage of the library is use the "options" function I mentioned above to select the correct encoding before connecting.

As workaround instead we used "search and replace" to add lines using SQL commands to configure the connection after every call to connect done with LuaSQL.

elouet commented 7 years ago

@speeder thanks so much for posting. I had the same issue exactly and I was able to fix it with a 1 line patch hard-coding "utf8" as charset before connecting. Now the c3 a9 does not get mangled any more.

This should be fixed at least in Debian because it sets the mysql default encoding to UTF-8.