Most of the time "FETCH BOOK" does nothing at all

[57194]

Environment:

$ docker images
REPOSITORY                             TAG             IMAGE ID       CREATED         SIZE
wabayang/jelu                          0.56.1          803001216fce   2 months ago    1.24GB

  ########
  # Jelu #
  jelu:
    image: "wabayang/jelu:0.56.1"
    container_name: "jelu"
    restart: "unless-stopped"
    ports: [ "127.0.0.1:11111:11111" ]
    volumes:
      - "./jelu/config:/config"
      - "./jelu/database:/database"
      - "./jelu/files/images:/files/images"
      - "./jelu/files/imports:/files/imports"
      - "/etc/timezone:/etc/timezone:ro"
    env_file: "./envs/jelu.env"
  # Jelu #
  ########

SPRING_DATASOURCE_USERNAME=redacted
SPRING_DATASOURCE_PASSWORD=redacted
JELU_AUTH_LDAP_ENABLED=false
JELU_AUTH_PROXY_ENABLED=true
JELU_AUTH_PROXY_ADMINNAME=redacted
JELU_AUTH_PROXY_HEADER=redacted
JELU_METADATA_CALIBRE_PATH=/calibre/fetch-ebook-metadata

---

Reproduction:

1. Go to /add-book
2. Click 🪄 AUTO FILL
3. Paste an ISBN into the Isbn field, e.g. 9781250186928
4. Click FETCH BOOK

Most of the time: nothing happens except a spinning circle and a green bar moving. Only once have I had this succeed on dozens and dozens of attempts with different ISBNs.

When performing Step 4, I see a 0 byte POST request is made to /api/v1/metadata and nothing at all appears in the jelu.log file nor in docker logs jelu.

The one time it succeeded, I did see a log entry like:

2024-07-31T13:55:02.646-05:00 DEBUG 1 --- [io-11111-exec-5] i.g.b.jelu.service.metadata.OpfParser    : parsed dto MetadataDto(title=All Systems Red, isbn10=null, isbn13=9780765397539, summary=<p>A New York Times and USA Today BestsellerWinner: 2018 Hugo Award for Best NovellaWinner: 2018 Nebula Award for Best NovellaWinner: 2018 Alex AwardWinner: 2018 Locus AwardOne of the Verge's Best Books of 2017A murderous android discovers itself in All Systems Red, a tense science fiction adventure by Martha Wells that interrogates the roots of consciousness through Artificial Intelligence.</p>
<p>"As a heartless killing machine, I was a complete failure.</p>
<p>"In a corporate-dominated spacefaring future, planetary missions must be approved and supplied by the Company. Exploratory teams are accompanied by Company-supplied security androids, for their own safety.</p>
<p>But in a society where contracts are awarded to the lowest bidder, safety isn’t a primary concern.</p>
<p>On a distant planet, a team of scientists are conducting surface tests, shadowed by their Company-supplied ‘droid — a self-aware SecUnit that has hacked its own governor module, and refers to itself (though never out loud) as “Murderbot.” Scornful of humans, all it really wants is to be left alone long enough to figure out who it is.</p>
<p>But when a neighboring mission goes dark, it's up to the scientists and their Murderbot to get to the truth.</p>
<p>The Murderbot DiariesAll Systems RedArtificial ConditionRogue ProtocolExit StrategyNetwork EffectFugitive TelemetrySystem Collapse</p>, image=null, publisher=Tor.com, pageCount=null, publishedDate=2017-05-02T07:00:00+00:00, authors=[Martha Wells], tags=[Science Fiction, Adult, Fantasy], series=The Murderbot Diaries, numberInSeries=1.0, language=eng, googleId=Zl5s0AEACAAJ, amazonId=0765397536, goodreadsId=33387769)
2024-07-31T13:58:10.128-05:00 DEBUG 1 --- [o-11111-exec-10] i.g.bayang.jelu.service.BookService      : renaming of metadata imported file meta-import-9780765397539-1722452096181.jpg was successful: true

Unfortunately, I didn't catch what the POST request looked like in my browser console that time.

If I manually run something like docker exec -it jelu /calibre/fetch-ebook-metadata -i 9781250186928 I can see an output almost immediately.

[57194]

Spammed tons of different ISBNs until one finally worked again. It was the one in the above text. That one failed literally dozens of times and now finally did something.

When it succeeded I had a 1.26KB POST request to the aforementioned API endpoint. So I don't know why most of the time an empty POST request is being sent.

[57194]

Now it's pretty consistently going…for the moment…

[bayang]

This is a known problem, the tool used to fetch metadata is severely rate limited by google. See #59

So I don't know why most of the time an empty POST request is being sent.

Are you sure an empty POST is sent or is it rather that the request is not empty, bu the backend timeouts calling the metadata fetcher tool and then returns an empty response to the frontend ?

When I have some time I'll add other metadata sources to not have to rely on this tool.

You can already use the google books api as a metadata source and not have this problem. see "Adding other metadata providers" at https://bayang.github.io/jelu-web/configuration/index.html

[57194]

This is a known problem, the tool used to fetch metadata is severely rate limited by google. See #59

I see. Noted.

So I don't know why most of the time an empty POST request is being sent.

Are you sure an empty POST is sent or is it rather that the request is not empty, bu the backend timeouts calling the metadata fetcher tool and then returns an empty response to the frontend ?

I guess the problem is the request never times out ever, so it just reads as 0 bytes? Dunno.

You can already use the google books api as a metadata source and not have this problem. see "Adding other metadata providers" at https://bayang.github.io/jelu-web/configuration/index.html

Is it possible to configure this via environment variables instead of a YAML config file?

[bayang]

You can already use the google books api as a metadata source and not have this problem. see "Adding other metadata providers" at https://bayang.github.io/jelu-web/configuration/index.html

Is it possible to configure this via environment variables instead of a YAML config file?

It should be possible. Jelu uses spring boot config format so any config can be passed through env vars. There are naming conventions though : https://docs.spring.io/spring-boot/reference/features/external-config.html#features.external-config.typesafe-configuration-properties.relaxed-binding

Since configuring plugins in jelu works with lists I found an explanation, you can try it :

see first answer here : https://stackoverflow.com/questions/55106571/environment-variables-for-list-in-spring-boot-configuration

tldr :

properties:
  topics:
    - 
      name: topic-01
      id: id-1
    - 
      name: topic-02
      id: id-2
    - 
      name: topic-03
      id: id-3

becomes in env vars :

PROPERTIES_TOPICS_0_NAME=topic-01
PROPERTIES_TOPICS_0_ID=id-01
PROPERTIES_TOPICS_1_NAME=topic-02
PROPERTIES_TOPICS_1_ID=id-02
PROPERTIES_TOPICS_2_NAME=topic-03
PROPERTIES_TOPICS_2_ID=id-03

[57194]

Awesome! Thanks. :-)

I'll close this one out since #59 already exists anyway.