Fix detection of encoding again

[benoit74]

Fix #312

Changes

• first try to find charset in document first 1024 bytes with the ascii or UTF-16 or UTF-32 encoding and regex ; if found, decode with this encoding in 'replace' mode (always going to work, do not care if encoding specified was wrong, not our fault)
  • NOTA: or UTF-16 or UTF-32 is a slight change compared to issue proposal because with UTF-16 and UTF-32 first bytes cannot be decoded to ascii properly (they always use at least 16 bytes or 32 bytes even for ascii characters)
• if no charset in found in document first 1024 bytes, search for charset in HTTP Content-Type header ; if found, decode with this encoding in 'replace' mode (always going to work, do not care if encoding specified was wrong, not our fault)
• if no charset found in document or HTTP header, we will have to guess a little bit (situation is quite common for JS and CSS); try UTF-8 in 'strict' mode ; if it fails, try ISO-8859-1 in 'strict' mode ; if it fails, stop the scraper (we cannot guess encoding when not specified in document or HTTP header) ; this list of charset to try should even probably be exposed as a scraper parameter --guessing-charsets so that it could be possible when needed to tweak this list when needed

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 93.75000% with 1 line in your changes missing coverage. Please review.

Project coverage is 84.49%. Comparing base (4c12681) to head (9933304).

:exclamation: Current head 9933304 differs from pull request most recent head b1c8a35

Please upload reports for the commit b1c8a35 to get more accurate results.

Files	Patch %	Lines
src/warc2zim/utils.py	91.66%	1 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #314 +/- ## ========================================== + Coverage 84.06% 84.49% +0.42% ========================================== Files 14 14 Lines 1268 1238 -30 Branches 249 245 -4 ========================================== - Hits 1066 1046 -20 + Misses 155 149 -6 + Partials 47 43 -4 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[benoit74]

CodeFactor is complaining about files which should be ignored since they are outside our scope, these are sample files from online websites.

[rgaudin]

Can you add exceptions then? Either inline, via a config file or via Codefactor UI

[benoit74]

Can we rename guessed_charsets with charsets_to_try ?

Sure

Good luck with Codefactor 😀

I will kill it ^^ It worked once, I supposed my last modification was ok so I cleaned up everything I've left over, it's not working anymore, I rolled-back to what was working, it is not working anymore. But I will nail it ^^

[benoit74]

Codefactor issues its configuration from main branch, so you have to merge first to main before you can review the PR ...

[rgaudin]

Go ahead

[kelson42]

@benoit74 Bravo... and bon courage!

[benoit74]

Go ahead

I already pushed only the configuration to main branch. My comment was more for "the posterity". I still have to rewrite this branch to change arg name + simplify commits.

Bravo... and bon courage!

We will finish by nailing this down ^^