Some websites, especially older ones, sometimes use a different encoding scheme than utf8 or latin1. Before, we simply tried different encoding schemes until one successfully decoded the bytes but this approach can fail unexpectedly as some encodings can erroneously get decoded by other encodings without errors being reported. We now use the encoding detection crate 'chardetng' which also seems to be the one used in firefox.
Closes #137.
Some websites, especially older ones, sometimes use a different encoding scheme than utf8 or latin1. Before, we simply tried different encoding schemes until one successfully decoded the bytes but this approach can fail unexpectedly as some encodings can erroneously get decoded by other encodings without errors being reported. We now use the encoding detection crate 'chardetng' which also seems to be the one used in firefox.