Closed panaC closed 4 years ago
Source URL:
http://gallica.bnf.fr/ark:/12148/bpt6k65669461.epub
Fetched filename:
Contes_fantastiques_(3e_éd_)_[...]Erckmann-Chatrian_Auteur_bpt6k65669461.epub
(1.2MB)
I can reproduce the fatal error:
Uncaught Exception:
Error [ERR_STREAM_WRITE_AFTER_END]: write after end
at writeAfterEnd (_stream_writable.js:243:12)
at WriteStream.Writable.write (_stream_writable.js:291:5)
at IncomingMessage.response.on (/Volumes/500GB/Code/readium-desktop/dist/main.js:5476:38)
at IncomingMessage.emit (events.js:187:15)
at addChunk (_stream_readable.js:279:12)
at readableAddChunk (_stream_readable.js:264:11)
at IncomingMessage.Readable.push (_stream_readable.js:219:10)
at HTTPParser.parserOnBody (_http_common.js:124:22)
at TLSSocket.socketOnData (_http_client.js:432:20)
at TLSSocket.emit (events.js:182:13)
at addChunk (_stream_readable.js:279:12)
at readableAddChunk (_stream_readable.js:264:11)
at TLSSocket.Readable.push (_stream_readable.js:219:10)
at TLSWrap.onread (net.js:636:20)
[2] readium-desktop:sync ### action type API_REQUEST +1ms
[2] readium-desktop:main#redux/sagas/api publication importOpdsEntry [ null,
[2] 'eyJNZXRhZGF0YSI6eyJUaXRsZSI6IkNvbnRlcyBmYW50YXN0aXF1ZXMgKDNlIMOpZC4pIC8gcGFyIEVyY2ttYW5uLUNoYXRyaWFuLi4uIiwiSWRlbnRpZmllciI6Imh0dHA6Ly9nYWxsaWNhLmJuZi5mci9hcms6LzEyMTQ4L2JwdDZrNjU2Njk0NjEiLCJNb2RpZmllZCI6IjIwMTktMDMtMDNUMDA6MDA6MDAuMDAwWiIsIlB1Ymxpc2hlciI6W3siTmFtZSI6InB1Ymxpc2hlciJ9XSwiQXV0aG9yIjpbeyJOYW1lIjoiRXJja21hbm4tQ2hhdHJpYW4gIiwiSWRlbnRpZmllciI6Imh0dHBzOi8vZ2FsbGljYS5ibmYuZnIvb3Bkcz9xdWVyeT1kYy5jcmVhdG9yIGFsbCBcIkVyY2ttYW5uLUNoYXRyaWFuIFwiIn1dLCJEZXNjcmlwdGlvbiI6IkNvbnRpZW50IHVuZSB0YWJsZSBkZXMgbWF0acOocmVzQXZlYyBtb2RlIHRleHRlIn0sIkxpbmtzIjpbeyJIcmVmIjoiaHR0cDovL2dhbGxpY2EuYm5mLmZyL2FyazovMTIxNDgvYnB0Nms2NTY2OTQ2MSIsIlR5cGVMaW5rIjoidGV4dC9odG1sIiwiUmVsIjpbImFsdGVybmF0ZSJdLCJUaXRsZSI6IlZvaXIgc3VyIEdhbGxpY2EifSx7IkhyZWYiOiJodHRwOi8vZ2FsbGljYS5ibmYuZnIvYXJrOi8xMjE0OC9icHQ2azY1NjY5NDYxLmVwdWIiLCJUeXBlTGluayI6ImFwcGxpY2F0aW9uL2VwdWIremlwIiwiUmVsIjpbImh0dHA6Ly9vcGRzLXNwZWMub3JnL2FjcXVpc2l0aW9uIl0sIlRpdGxlIjoiT3V2cmlyIGxlIGxpdnJlIn1dLCJJbWFnZXMiOlt7IkhyZWYiOiJodHRwOi8vZ2FsbGljYS5ibmYuZnIvYXJrOi8xMjE0OC9icHQ2azY1NjY5NDYxLnRodW1ibmFpbCIsIlR5cGVMaW5rIjoiaW1hZ2UvanBlZyIsIlJlbCI6WyJodHRwOi8vb3Bkcy1zcGVjLm9yZy9pbWFnZS90aHVtYm5haWwiXX0seyJIcmVmIjoiaHR0cDovL2dhbGxpY2EuYm5mLmZyL2FyazovMTIxNDgvYnB0Nms2NTY2OTQ2MS5oaWdocmVzIiwiVHlwZUxpbmsiOiJpbWFnZS9qcGVnIiwiUmVsIjpbImh0dHA6Ly9vcGRzLXNwZWMub3JnL2ltYWdlIl19XX0=',
[2] 'Contes fantastiques (3e éd.) / par Erckmann-Chatrian...',
[2] null ] +19ms
[2] readium-desktop:sync ### action type DOWNLOAD_REQUEST +0ms
[2] readium-desktop:sync ### action type TOAST_OPEN_REQUEST +1ms
[2] readium-desktop:main#services/catalog [START] Download publication http://gallica.bnf.fr/ark:/12148/bpt6k65669461.epub +0ms
[2] readium-desktop:main#services/downloader Error while downloading resource { identifier: 'f8319469-996c-4d19-8619-70e992eab631',
[2] srcUrl: 'http://gallica.bnf.fr/ark:/12148/bpt6k65669461.epub',
[2] dstPath:
[2] '/var/folders/f4/bs_1cm7565jdq6tzs0n01h_40000gn/T/readium-desktop-166112MHtU6bwhYnm.epub.part',
[2] progress: 0,
[2] downloadedSize: 0,
[2] status: 3 } 500 +0ms
[2] readium-desktop:main#redux/sagas/api Error while downloading resource +528ms
[2] readium-desktop:sync ### action type API_ERROR +527ms
[2] readium-desktop:renderer:bookshelf_ Error to fetch api publication/importOpdsEntry undefined +8s
We are getting HTTP status code 500
when downloading from Thorium, but it works from the web browser. So I suspect our HTTP request headers are missing something that the Galica server wants (e.g. user-agent, origin, etc.).
https://github.com/readium/readium-desktop/blob/e65e2a5ce848aae05119148d85592989a553a078/src/main/services/downloader.ts#L93-L103
Note that I think there is also a bug in the if (response.statusCode < 200 || response.statusCode > 299)
conditional statement: return
is missing!! (otherwise response.on("data")
will run!)
curl -s -L -I -X GET http://gallica.bnf.fr/ark:/12148/bpt6k65669461.epub
HTTP/1.1 301 Moved Permanently
Date: Fri, 27 Sep 2019 22:03:04 GMT
Server: Apache
Location: https://gallica.bnf.fr/ark:/12148/bpt6k65669461.epub
Content-Length: 260
Content-Type: text/html; charset=iso-8859-1
HTTP/2 200
date: Fri, 27 Sep 2019 22:03:05 GMT
server: Apache
set-cookie: JSESSIONID=1B35E5BA6697AFE3EE2EB8C408A83545; Path=/; Secure; HttpOnly
content-disposition: inline;filename="Contes_fantastiques_(3e_?d_)_[...]Erckmann-Chatrian_Auteur_bpt6k65669461.epub"
content-type: application/epub+zip;charset=UTF-8
content-language: fr-FR
content-length: 1211216
vary: Accept-Encoding,User-Agent
HTTP2:
curl -s -L -v https://gallica.bnf.fr/ark:/12148/bpt6k65669461.epub
* Trying 194.199.8.11...
* TCP_NODELAY set
* Connected to gallica.bnf.fr (194.199.8.11) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=FR; ST=Ile de France; L=Paris; O=Bibliotheque Nationale de France; CN=*.bnf.fr
* start date: May 28 00:00:00 2018 GMT
* expire date: Jun 1 12:00:00 2020 GMT
* subjectAltName: host "gallica.bnf.fr" matched cert's "*.bnf.fr"
* issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fd20a006600)
> GET /ark:/12148/bpt6k65669461.epub HTTP/2
> Host: gallica.bnf.fr
> User-Agent: curl/7.54.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200
< date: Fri, 27 Sep 2019 22:04:34 GMT
< server: Apache
< set-cookie: JSESSIONID=B1DF9D367BC95DAD2373C307E98A60F6; Path=/; Secure; HttpOnly
< content-disposition: inline;filename="Contes_fantastiques_(3e_?d_)_[...]Erckmann-Chatrian_Auteur_bpt6k65669461.epub"
< content-type: application/epub+zip;charset=UTF-8
< content-language: fr-FR
< content-length: 1211216
< vary: Accept-Encoding,User-Agent
Minimal repro test case: https://repl.it/languages/nodejs with:
const request = require("request");
const requestStream = request.get(" https://gallica.bnf.fr/ark:/12148/bpt6k65669461.epub",{timeout: 5000});
requestStream.on("error", (error) => { console.log(error); });
requestStream.on("response", (response) => { console.log(response.statusCode); });
=> 500
As I suspected, the missing User-Agent
HTTP header is the cause.
Repro:
https://repl.it/languages/nodejs
const request = require("request");
const requestStream = request.get("https://gallica.bnf.fr/ark:/12148/bpt6k65669461.epub", {
timeout: 5000,
headers: {
'User-Agent': 'Thorium'
}
});
// requestStream.on("error", (error) => { console.log(error); });
requestStream.on("response", (response) => { console.log(response.statusCode); });
=> 200
feed :: bnf.fr (gallica): "http://gallica.bnf.fr/opds"
click on
derniere mise en ligne
: