search

edrlab / thorium-reader

A cross platform desktop reading app, based on the Readium Desktop toolkit
https://www.edrlab.org/software/thorium-reader/
BSD 3-Clause "New" or "Revised" License
1.73k stars 151 forks source link

Thorium Reader is not able to borrow books from an OPDS feed #1148

Closed vbessonov closed 3 years ago

vbessonov commented 4 years ago

I faced with an issue when Thorium Reader is not able to borrow books from an OPDS feed.

The OPDS feed I'm using: https://github.com/vbessonov/opds-feed

Steps to reproduce:

  1. git clone https://github.com/vbessonov/opds-feed
  2. cd opds-feed
  3. docker-compose up -d
  4. Open Thorium Reader and add a new OPDS feed using http://localhost:8080 as its URL: image
  5. Open Test OPDS feed catalog and try to borrow a book: image
  6. Thorium Reader opens a borrow link in a browser and then stops: image

The books wasn't downloaded and isn't accessible in Thorium Reader: image

danielweck commented 4 years ago

Thank you very much for the detailed description of the problem.

Thorium does not support PDF publications, so this could be the cause of the behaviour. Would you be able to test with an OPDS feed that serves EPUB publications?

vbessonov commented 4 years ago

Absolutely, will do

CircularKen commented 4 years ago

Should that be "Thorium does not yet support PDF publications"…?

vbessonov commented 4 years ago

@danielweck I tried with an EPUB and updated my repository accordingly but it worked the same way, I wasn't able to download the book

panaC commented 4 years ago

@vbessonov publish the log, please

vbessonov commented 4 years ago

@panaC, sure but I downloaded the Thorium's binary from the website, could you tell me how I can get the logs?

panaC commented 4 years ago

https://github.com/vbessonov/opds-feed/blob/15514f87702dff9d3195c03ed4fce8af4a3d315e/content/borrow/index.xml#L29

maybe try to replace href=http://localhost:8080/fulfill/ with http://localhost:8080/fulfill/index.epub

vbessonov commented 4 years ago

Unfortunately, it didn't work. Basically, http://localhost:8080/fulfill/ gets rewritten by nginx to http://localhost:8080/fulfill/index.epub anyways

vbessonov commented 4 years ago

@panaC, is there anything else I can do to make it work?

vbessonov commented 4 years ago

@danielweck, what can I do to speed up the investigation process?

danielweck commented 4 years ago

Sorry, I have been out of office. You could try running Thorium from its source code (develop branch): 1) git clone https://github.com/edrlab/thorium-reader.git 2) cd thorium-reader 3) npm install (this will take a while ... so grab your drink of choice) 4) npm run start:dev (or SKIP_CHECK_TYPESCRIPT=1 npm run start:dev to speed up the compilation process)

danielweck commented 4 years ago

So far I am not seeing anything suspicious in the OPDS server:

curl -s -L -I -X GET "http://localhost:8080/borrow" =>

HTTP/1.1 301 Moved Permanently
Server: nginx/1.19.2
Date: Fri, 21 Aug 2020 17:23:16 GMT
Content-Type: text/html
Content-Length: 169
Location: http://localhost/borrow/
Connection: keep-alive

curl -s -L -I -X GET "http://localhost:8080/borrow/index.xml" =>

HTTP/1.1 200 OK
Server: nginx/1.19.2
Date: Fri, 21 Aug 2020 17:23:30 GMT
Content-Type: application/atom+xml
Content-Length: 2931
Last-Modified: Fri, 21 Aug 2020 17:17:27 GMT
Connection: keep-alive
ETag: "5f4001a7-b73"
Accept-Ranges: bytes
danielweck commented 4 years ago

EDIT --- see message below (there is a typo in the OPDS feed)

@panaC I am seeing this in the console:

(note that I added http://localhost:8080/borrow/index.xml directly to the Catalogs, instead of http://localhost:8080/, and I also tried changing the href from http://localhost:8080/fulfill/ to http://localhost:8080/fulfill/index.epub in http://localhost:8080/borrow/index.xml)

[2]   readium-desktop:main#services/publication [START] Download publication http://localhost:8080/fulfill/index.epub +3m
[2]   readium-desktop:main#services/downloader DOWNLOAD process:  http://localhost:8080/fulfill/index.epub +3m
[2]   readium-desktop:main#services/downloader oauth get +1ms
[2]   readium-desktop:main#services/downloader NotFoundError { message: 'document not found' } +0ms
[2]   readium-desktop:main#services/downloader Error while downloading resource {
[2]   identifier: '1ce5431d-1ffd-4fec-b347-6f3b54331ba5',
[2]   srcUrl: 'http://localhost:8080/fulfill/index.epub',
[2]   dstPath: '/var/folders/gm/ws1_n9s97y1b3z4vfgw79h240000gn/T/readium-desktop--23667-5N4QZxgbEXv0-.epub',
[2]   extension: '.epub',
[2]   progress: 0,
[2]   downloadedSize: 0,
[2]   status: 3
[2] } 404 +6ms
[2]   readium-desktop:sync ### action type TOAST_OPEN_REQUEST +9ms
[2]   readium-desktop:sync send to 816323f2-4cc6-4497-b4f3-30267c081898 +0ms
[2]   readium-desktop:sync ### action type DOWNLOAD_ERROR +0ms
[2]   readium-desktop:sync send to 816323f2-4cc6-4497-b4f3-30267c081898 +0ms
[2]   readium-desktop:main:saga:api API-ERROR Error: importPublicationFromOpdsDoc error Error while downloading resource: 404
[2]     at PublicationApi.<anonymous> (/Users/danielweck/Downloads/500GB/Code/readium-desktop/dist/main.js:3162:27)
[2]     at Generator.throw (<anonymous>)
[2]     at rejected (/Users/danielweck/Downloads/500GB/Code/readium-desktop/node_modules/tslib/tslib.js:112:69)
[2]     at processTicksAndRejections (internal/process/task_queues.js:94:5) requestAction:  {
[2]   type: 'API_REQUEST',
[2]   payload: [
[2]     {
[2]       url: 'http://localhost:8080/fulfill/index.epub',
[2]       title: undefined,
[2]       type: 'application/epub+zip',
[2]       properties: undefined
[2]     },
[2]     'eyJtZXRhZGF0YSI6eyJ0aXRsZSI6IkRvc3RvZXZza3kgRXB1YiIsImlkZW50aWZpZXIiOiJ1cm46aXNibjo5NzgwMjMxNTQzOTczIiwiYXV0aG9yIjoiU3BlYyBXcml0ZXIiLCJsYW5ndWFnZSI6ImVuIiwibW9kaWZpZWQiOiIyMDIwLTA4LTE3VDE1OjQ4OjQxLjAwMFoiLCJwdWJsaXNoZWQiOiIyMDIwLTA4LTE3VDAwOjAwOjAwLjAwMFoiLCJzdWJqZWN0IjpbeyJuYW1lIjoiQWR1bHQiLCJzY2hlbWUiOiJodHRwOi8vc2NoZW1hLm9yZy9hdWRpZW5jZSIsImNvZGUiOiJBZHVsdCJ9LHsibmFtZSI6Ik5vbmZpY3Rpb24iLCJzY2hlbWUiOiJodHRwOi8vbGlicmFyeXNpbXBsaWZpZWQub3JnL3Rlcm1zL2ZpY3Rpb24vIiwiY29kZSI6Imh0dHA6Ly9saWJyYXJ5c2ltcGxpZmllZC5vcmcvdGVybXMvZmljdGlvbi9Ob25maWN0aW9uIn0seyJuYW1lIjoiVGVjaG5vbG9neSIsInNjaGVtZSI6Imh0dHA6Ly9saWJyYXJ5c2ltcGxpZmllZC5vcmcvdGVybXMvZ2VucmVzL1NpbXBsaWZpZWQvIiwiY29kZSI6Imh0dHA6Ly9saWJyYXJ5c2ltcGxpZmllZC5vcmcvdGVybXMvZ2VucmVzL1NpbXBsaWZpZWQvVGVjaG5vbG9neSJ9LHsibmFtZSI6IkFyY2hpdGVjdHVyZSIsInNjaGVtZSI6Imh0dHA6Ly9saWJyYXJ5c2ltcGxpZmllZC5vcmcvdGVybXMvZ2VucmVzL1NpbXBsaWZpZWQvIiwiY29kZSI6Imh0dHA6Ly9saWJyYXJ5c2ltcGxpZmllZC5vcmcvdGVybXMvZ2VucmVzL1NpbXBsaWZpZWQvQXJjaGl0ZWN0dXJlIn1dfSwibGlua3MiOlt7InJlbCI6Imlzc3VlcyIsImhyZWYiOiJodHRwOi8vY20uaGlsYmVydHRlYW0ubmV0OjY1MDAvTUFJTi93b3Jrcy9JU0JOLzk3ODAyMzE1NDM5NzMvcmVwb3J0In0seyJ0eXBlIjoiYXBwbGljYXRpb24vYXRvbSt4bWw7dHlwZT1lbnRyeTtwcm9maWxlPW9wZHMtY2F0YWxvZyIsInJlbCI6ImFsdGVybmF0ZSIsImhyZWYiOiJodHRwOi8vY20uaGlsYmVydHRlYW0ubmV0OjY1MDAvTUFJTi93b3Jrcy9JU0JOLzk3ODAyMzE1NDM5NzMifSx7InR5cGUiOiJhcHBsaWNhdGlvbi9lcHViK3ppcCIsInJlbCI6Imh0dHA6Ly9vcGRzLXNwZWMub3JnL2FjcXVpc2l0aW9uIiwiaHJlZiI6Imh0dHA6Ly9sb2NhbGhvc3Q6ODA4MC9mdWxmaWxsL2luZGV4LmVwdWIifSx7InJlbCI6Imh0dHA6Ly9saWJyYXJ5c2ltcGxpZmllZC5vcmcvdGVybXMvcmVsL3Jldm9rZSIsImhyZWYiOiJodHRwOi8vY20uaGlsYmVydHRlYW0ubmV0OjY1MDAvTUFJTi9sb2Fucy8xMC9yZXZva2UifSx7InR5cGUiOiJhcHBsaWNhdGlvbi9hdG9tK3htbDtwcm9maWxlPW9wZHMtY2F0YWxvZztraW5kPWFjcXVpc2l0aW9uIiwidGl0bGUiOiJSZWNvbW1lbmRlZCBXb3JrcyIsInJlbCI6InJlbGF0ZWQiLCJocmVmIjoiaHR0cDovL2NtLmhpbGJlcnR0ZWFtLm5ldDo2NTAwL01BSU4vd29ya3MvSVNCTi85NzgwMjMxNTQzOTczL3JlbGF0ZWRfYm9va3MifSx7InR5cGUiOiJhcHBsaWNhdGlvbi9sZCtqc29uOyBwcm9maWxlPVwiaHR0cDovL3d3dy53My5vcmcvbnMvYW5uby5qc29ubGRcIiIsInJlbCI6Imh0dHA6Ly93d3cudzMub3JnL25zL29hI2Fubm90YXRpb25TZXJ2aWNlIiwiaHJlZiI6Imh0dHA6Ly9jbS5oaWxiZXJ0dGVhbS5uZXQ6NjUwMC9NQUlOL2Fubm90YXRpb25zL0lTQk4vOTc4MDIzMTU0Mzk3MyJ9LHsicmVsIjoiaHR0cDovL2xpYnJhcnlzaW1wbGlmaWVkLm9yZy90ZXJtcy9yZWwvYW5hbHl0aWNzL29wZW4tYm9vayIsImhyZWYiOiJodHRwOi8vY20uaGlsYmVydHRlYW0ubmV0OjY1MDAvTUFJTi9hbmFseXRpY3MvSVNCTi85NzgwMjMxNTQzOTczL29wZW5fYm9vayJ9XX0='
[2]   ],
[2]   meta: {
[2]     api: {
[2]       requestId: 'SAME_FILE_IMPORT_REQUEST',
[2]       moduleId: 'publication',
[2]       methodId: 'importOpdsPublicationLink'
[2]     }
[2]   },
[2]   sender: { type: 1, identifier: '816323f2-4cc6-4497-b4f3-30267c081898' }
[2] } +10ms
danielweck commented 4 years ago

@vbessonov there is a typo in:

https://github.com/vbessonov/opds-feed/blob/15514f87702dff9d3195c03ed4fce8af4a3d315e/content/borrow/index.xml#L29

fulfil vs. fulfill

curl -s -L -I -X GET "http://localhost:8080/fulfil/index.epub" =>

HTTP/1.1 200 OK
Server: nginx/1.19.2
Date: Fri, 21 Aug 2020 17:51:51 GMT
Content-Type: application/epub+zip
Content-Length: 798536
Last-Modified: Fri, 21 Aug 2020 17:17:27 GMT
Connection: keep-alive
ETag: "5f4001a7-c2f48"
Accept-Ranges: bytes
danielweck commented 4 years ago

@panaC when clicking on the EPUB cover, the popup dialog appears and shows the "borrow" link, and a couple of seconds later this gets replaced with an "import" link (presumably, there is an async HTTP fetch to a "publication" dataset, which replaces the default one found in the original OPDS feed). The "import" link works fine. During the 2 seconds gap, if I click on the "borrow" link, it can be opened in the web browser, but not in Thorium. Any idea what is going on?

panaC commented 3 years ago 
  <entry schema:additionalType="http://schema.org/EBook">
    <title>Dostoevsky Epub</title>
    <author>
      <name>Spec Writer</name>
      <link href="http://cm.hilbertteam.net:6500/MAIN/works/contributor/Spec%20Writer/eng/Adult%2CAdults%2BOnly%2CAll%2BAges%2CChildren%2CYoung%2BAdult" type="application/atom+xml;profile=opds-catalog;kind=acquisition" rel="contributor" title="Spec Writer"/>
    </author>
    <simplified:pwid>1ef566f9-bda4-109d-e475-729437d2b6da</simplified:pwid>
    <category term="Adult" scheme="http://schema.org/audience" label="Adult"/>
    <category term="http://librarysimplified.org/terms/fiction/Nonfiction" scheme="http://librarysimplified.org/terms/fiction/" label="Nonfiction"/>
    <category term="http://librarysimplified.org/terms/genres/Simplified/Technology" scheme="http://librarysimplified.org/terms/genres/Simplified/" label="Technology"/>
    <category term="http://librarysimplified.org/terms/genres/Simplified/Architecture" scheme="http://librarysimplified.org/terms/genres/Simplified/" label="Architecture"/>
    <dcterms:language>en</dcterms:language>
    <link href="http://cm.hilbertteam.net:6500/MAIN/works/ISBN/9780231543973/report" rel="issues"/>
    <id>urn:isbn:9780231543973</id>
    <link href="http://cm.hilbertteam.net:6500/MAIN/works/ISBN/9780231543973" type="application/atom+xml;type=entry;profile=opds-catalog" rel="alternate"/>
    <bibframe:distribution bibframe:ProviderName="oapen"/>
    <published>2020-08-17T00:00:00Z</published>
    <link href="http://cm.hilbertteam.net:6500/MAIN/groups/142?entrypoint=Book" rel="collection" title="Nonfiction"/>
    <updated>2020-08-17T15:48:41Z</updated>
    <link href="http://localhost:8080/borrow/" rel="http://opds-spec.org/acquisition/borrow" type="application/atom+xml;type=entry;profile=opds-catalog">
      <opds:indirectAcquisition type="application/epub+zip"/>
      <opds:availability status="available"/>
    </link>
    <link href="http://cm.hilbertteam.net:6500/MAIN/works/ISBN/9780231543973/related_books" type="application/atom+xml;profile=opds-catalog;kind=acquisition" rel="related" title="Recommended Works"/>
    <link href="http://cm.hilbertteam.net:6500/MAIN/annotations/ISBN/9780231543973" type="application/ld+json; profile=&quot;http://www.w3.org/ns/anno.jsonld&quot;" rel="http://www.w3.org/ns/oa#annotationService"/>
    <link href="http://cm.hilbertteam.net:6500/MAIN/analytics/ISBN/9780231543973/open_book" rel="http://librarysimplified.org/terms/rel/analytics/open-book"/>
  </entry>

2 entry links are extracted from this entry : 

[2]   {
[2]     url: 'http://cm.hilbertteam.net:6500/MAIN/works/ISBN/9780231543973',
[2]     title: undefined,
[2]     type: 'application/atom+xml;type=entry;profile=opds-catalog',
[2]     properties: undefined
[2]   },
[2]   {
[2]     url: 'http://localhost:8080/borrow/',
[2]     title: undefined,
[2]     type: 'application/atom+xml;type=entry;profile=opds-catalog',
[2]     properties: {
[2]       indirectAcquisitionType: 'application/epub+zip',
[2]       lcpHashedPassphrase: undefined,
[2]       numberOfItems: undefined,
[2]       priceValue: undefined,
[2]       priceCurrency: undefined,
[2]       holdTotal: undefined,
[2]       holdPosition: undefined,
[2]       copyTotal: undefined,
[2]       copyAvailable: undefined,
[2]       availabilityState: undefined,
[2]       availabilitySince: undefined,
[2]       availabilityUntil: undefined
[2]     }
[2]   }

the first timeout and the second returns : 

<entry xmlns:app="http://www.w3.org/2007/app"
  xmlns:bib="http://bib.schema.org/"
  xmlns:bibframe="http://bibframe.org/vocab/"
  xmlns:dcterms="http://purl.org/dc/terms/"
  xmlns:drm="http://librarysimplified.org/terms/drm"
  xmlns:opds="http://opds-spec.org/2010/catalog"
  xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/"
  xmlns:opf="http://www.idpf.org/2007/opf"
  xmlns:schema="http://schema.org/"
  xmlns:simplified="http://librarysimplified.org/terms/"
  xmlns="http://www.w3.org/2005/Atom" schema:additionalType="http://schema.org/EBook">
  <title>Dostoevsky Epub</title>
  <author>
    <name>Spec Writer</name>
    <link href="http://cm.hilbertteam.net:6500/MAIN/works/contributor/Spec%20Writer/eng/Adult%2CAdults%2BOnly%2CAll%2BAges%2CChildren%2CYoung%2BAdult" type="application/atom+xml;profile=opds-catalog;kind=acquisition" rel="contributor" title="Spec Writer"/>
  </author>
  <simplified:pwid>1ef566f9-bda4-109d-e475-729437d2b6da</simplified:pwid>
  <category term="Adult" scheme="http://schema.org/audience" label="Adult"/>
  <category term="http://librarysimplified.org/terms/fiction/Nonfiction" scheme="http://librarysimplified.org/terms/fiction/" label="Nonfiction"/>
  <category term="http://librarysimplified.org/terms/genres/Simplified/Technology" scheme="http://librarysimplified.org/terms/genres/Simplified/" label="Technology"/>
  <category term="http://librarysimplified.org/terms/genres/Simplified/Architecture" scheme="http://librarysimplified.org/terms/genres/Simplified/" label="Architecture"/>
  <dcterms:language>en</dcterms:language>
  <link href="http://cm.hilbertteam.net:6500/MAIN/works/ISBN/9780231543973/report" rel="issues"/>
  <id>urn:isbn:9780231543973</id>
  <link href="http://cm.hilbertteam.net:6500/MAIN/works/ISBN/9780231543973" type="application/atom+xml;type=entry;profile=opds-catalog" rel="alternate"/>
  <bibframe:distribution bibframe:ProviderName="oapen"/>
  <published>2020-08-17T00:00:00Z</published>
  <updated>2020-08-17T15:48:41Z</updated>
  <link href="http://localhost:8080/fulfill/" rel="http://opds-spec.org/acquisition" type="application/epub+zip">
    <opds:availability since="2020-08-17T15:52:17Z" status="available" until="2020-09-07T15:52:17Z"/>
  </link>
  <link href="http://cm.hilbertteam.net:6500/MAIN/loans/10/revoke" rel="http://librarysimplified.org/terms/rel/revoke"/>
  <link href="http://cm.hilbertteam.net:6500/MAIN/works/ISBN/9780231543973/related_books" type="application/atom+xml;profile=opds-catalog;kind=acquisition" rel="related" title="Recommended Works"/>
  <link href="http://cm.hilbertteam.net:6500/MAIN/annotations/ISBN/9780231543973" type="application/ld+json; profile=&quot;http://www.w3.org/ns/anno.jsonld&quot;" rel="http://www.w3.org/ns/oa#annotationService"/>
  <link href="http://cm.hilbertteam.net:6500/MAIN/analytics/ISBN/9780231543973/open_book" rel="http://librarysimplified.org/terms/rel/analytics/open-book"/>
</entry>

the publication information view is updated and display this new entry. There is no more a borrow link but an acquisition link. This is why the view changed. as far as borrow link is concerned, it's not an acceptable mimeType to download it and therefore opens the web browser.

the acceptable mimeType are : 

const supportedFileTypeLinkArray = [
    ContentType.AudioBookPacked,
    ContentType.AudioBookPackedLcp,
    ContentType.Epub,
    ContentType.Lcp,
    ContentType.AudioBook,
    ContentType.Json,
    ContentType.JsonLd,
];

json and jsonld are only accepted when the format is readium web publication manifest (RWPM) to package it.

I think this behaviour is standard, and i can close this issue. feel free to reopen it.

llemeurfr commented 3 years ago

@vbessonov is it finally something NYPL has to correct in its feeds?

panaC commented 3 years ago

My bad, It's an issue with opds1 . I think. Borrow link isn't an alternate opdsEntry.

https://github.com/edrlab/thorium-reader/blob/57b818376fbfe8be99362246abe66e7648c23b56/src/main/converter/opds.ts#L322-L325

What do you think @danielweck ?

panaC commented 3 years ago

I was wrong on the borrow that opens in the browser. The issue is here : https://github.com/edrlab/thorium-reader/blob/ac9595581b6c3878d5f8aaabf96cf8eb9351c9a9/src/renderer/library/opds/handleLink.ts#L28 and will be probably fixed with https://github.com/edrlab/thorium-reader/blob/b9426c75d376a59b6780a5ea7d61292caa327470/src/renderer/library/opds/handleLink.ts#L25-L29

panaC commented 3 years ago

tested with : https://github.com/vbessonov/opds-feed ,, it's works now !