HELP!! when importing csv line by line, encoding is wrong

[thierrydelbart]

Please help me! I worked on this for a few hours without success.

How to reproduce:

1. Change max_parsing_lines to 10 so the sync goes faster
2. Call /sync route, this will clear the base and import the first 10 items You should see the encoding problems calling /:base_slug/item

[watter]

You should take a look at your csv file encoding.

Probably you're trying to import an iso-8859-1 encoded file which should result in that mess.

Take a look at: http://stackoverflow.com/questions/14551608/list-of-encodings-that-node-js-supports

Cheers,

LEslie

[thierrydelbart]

Thanks @watter for the advise. I did try your link, in fact that's one of the first things I did when the problem appeared, but I din't manage to make it work.

[watter]

Well, as you didn't write what you've properly done here you have a few things to check:

1. make sure that your csv file encoding is at the same encoding you're working with (utf8) -- you can do this by either converting the file with iconv prior to importing it, or by code as stated in the link above.
2. make sure that your browser is handling your output as utf8 -- that'll depend on your browser's config.
3. clear all your cache between tests -- eventually you should configure your browser's cache to 0 (zero). Cache's screwd up many tests before, so be sure you don't have it.

good luck

[thierrydelbart]

I think I got it! Thanks to great iconv-lite node module. Here is my corrections:

request({url: csv.url});
.pipe(iconv.decodeStream('ISO-8859-1'))
.pipe(iconv.encodeStream('utf8'))
.pipe(csvParser({separator: separator,raw: false}))

(I was sure I had tried this solution before but probably with different encodings :( ) I'm going to deploy it on server to see on production :) Thank to everyone that took a look at it @watter @FredySchaible @biketrooper