search

tsibley / App-cpangrep

Grep CPAN from the command-line using grep.cpan.me
Other
4 stars 1 forks source link

Error decoding JSON response: malformed UTF-8 character in JSON string ... #7

Open perlancar opened 7 years ago

perlancar commented 7 years ago

% cpangrep copyright_year Error decoding JSON response: malformed UTF-8 character in JSON string, at character offset 95954 (before "\x{fffd} Robert May ...") at /home/s1/perl5/perlbrew/perls/perl-5.24.0/lib/site_perl/5.24.0/App/cpangrep.pm line 125.

tsibley commented 7 years ago

Thanks for this report. It's a known issue, although one I'd forgotten about. The cpangrep web service returns strictly bytes, but the cpangrep program needs to deal in characters for presentation. Unfortunately, the encoding of the bytes from the web service varies from hit to hit as the files found by the search may be in different encodings! Assuming UTF-8 globally was a dead-easy (but not always right) default choice I made early on. A better approach might be to decode as UTF-8 but with replacement characters instead of fatal errors, leading to a "best effort" approach.

perlancar commented 7 years ago

Yup, the "best effort" approach (even with inappropriate characters or just '?') would generally be better for users than die-ing.