node-fetch / fetch-charset-detection

Charset detection and conversion, originally from node-fetch.
MIT License
11 stars 3 forks source link

Error upgrading from node-fetch 2.x to node-fetch 3.x + fetch-charset-detection #151

Closed danburzo closed 2 years ago

danburzo commented 3 years ago

I'm using the upgrade guide for node-fetch 3.x and this code:

import fetch from 'node-fetch';
import convertBody from 'fetch-charset-detection';

fetch(`http://www.zeno.org/Philosophie/M/Hegel,+Georg+Wilhelm+Friedrich/Enzyklop%C3%A4die+der+philosophischen+Wissenschaften+im+Grundrisse/Erster+Teil.+Die+Wissenschaft+der+Logik./2.+Abteilung%3A+Die+Lehre+vom+Wesen`).then(async res => {
    const buf = await res.arrayBuffer();
    const text = convertBody(buf, res.headers);
});

Produces this error:

(node:20518) ExperimentalWarning: stream/web is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
/Users/danburzo/projects/percollate/node_modules/iconv-lite/lib/index.js:104
                throw new Error("Encoding not recognized: '" + encoding + "' (searched as: '"+enc+"')");
                      ^

Error: Encoding not recognized: 'de' (searched as: 'de')
LinusU commented 3 years ago

This is in the source of that url:

    <meta http-equiv="Language" content="de" />
    <meta http-equiv="Content-Language" content="de" />
    <meta http-equiv="Content-Type" content="text/html;charset=iso-8859-1" />

This is where it picks the wrong meta tag:

https://github.com/node-fetch/fetch-charset-detection/blob/05433e57f54e675f82ff9769e80c029e07c5d2ec/source/utils/get-charset.ts#L31