BaseAdresseNationale / api-geocode

API haute performance de géocodage CSV
MIT License
10 stars 3 forks source link

Switch to jschardet #13

Closed poudro closed 3 years ago

poudro commented 3 years ago

The PR replaces jschardet-french with jschardet as it is now outdated and improperly detects utf8 as latin1 in some cases.

For example:

const iconv = require("iconv-lite");
const frchardet = require("jschardet-french");

console.log(frchardet.detect(iconv.encode("É", "utf8"))); 

erroneously returns { encoding: 'windows-1252', confidence: 0.95 }

Whereas

const iconv = require("iconv-lite");
const chardet = require("jschardet");

console.log(chardet.detect(iconv.encode("É", "utf8"))); 

correctly returns { encoding: 'UTF-8', confidence: 0.99 }

jdesboeufs commented 3 years ago

Thank you very much! Can you update the yarn.lock file running yarn command? The actual file contains a reference to jschardet-french.

poudro commented 3 years ago

@jdesboeufs it's updated, the reference to jschardet-french comes from addok-geocode-stream (if I'm not mistaken)

jdesboeufs commented 3 years ago

Deployed in production 🚀

poudro commented 3 years ago

thanks @jdesboeufs !