search

gonzus / JavaScript-Duktape-XS

Perl XS binding for the Duktape JavaScript engine
MIT License
4 stars 5 forks source link

Wrong enconding in the final output #9

Closed Askoth closed 6 years ago

Askoth commented 6 years ago

We know the the encoding of the data is fine before sending it to JS. We know that the final output is wrong.

Somewhere in the middle it seems to be expecing latin instead of utf8 and is messing up.

eg: displaying Calendário instead of Calendário

gonzus commented 6 years ago

Good catch, thanks @Askoth. I just uploaded a new version 0.52 which fixes this (and adds a unit test). I still need to fix the same issue for strings used as hash keys (which require a bit of a different treatment), but strings anywhere else should be treated now as utf8.

I will keep the issue open until I fix the hash keys thing.

Askoth commented 6 years ago

I realized this morning that we could force utf8 by adding a second parameter to the get.

What do you think? We wouldn’t have problems with binaries. (If we ever need it)

Inspired in nodejs (file system)[https://nodejs.org/api/fs.html]:

duktape->get(‘something’, { encoding: ‘utf8’ })

Would this be bad considering we are changing duktape api? Maybe we need another method?

gonzus commented 6 years ago

I considered what you propose, and decided to go the easy way now and treat strings as UTF8 always. If we need more flexibility, we can later add a parameter that requests no encoding at all, so that the default caters for the most common case.

On Mon, 14 May 2018, 07:30 Marcelo Oliveira, notifications@github.com wrote:

I realized this morning that we could force utf8 by adding a second parameter to the get.

What do you think? We wouldn’t have problems with binaries. (If we ever need it)

Inspired in nodejs (file system)[https://nodejs.org/api/fs.html]:

duktape->get(‘something’, { encoding: ‘utf8’ })

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/gonzus/JavaScript-Duktape-XS/issues/9#issuecomment-388701349, or mute the thread https://github.com/notifications/unsubscribe-auth/AAMuogJHI6pBlM6bN0canJ2N7Sl-b3fPks5tyRZugaJpZM4T2k72 .

gonzus commented 6 years ago

I just fixed the hash keys thing, so now we can roundtrip hashes whose keys contain UTF8 characters => ca871841f7288330091331b341a34709a3a67a63.