search

szarroug3 / X-Ray_Calibre_Plugin

X-Ray Creator plugin for Calibre
http://www.mobileread.com/forums/showthread.php?t=273189
GNU General Public License v3.0
57 stars 12 forks source link

Books lacking description, other titles by author, etc. #73

Closed stoduk closed 8 years ago

stoduk commented 8 years ago

For the few books I've tried I find that the books are lacking any information such as description, other titles by the author, etc. I also found that the AuthorProfile file contained multiple copies of the same author info (eg. for a book with two authors I found the same two authors repated three times, for a total of six entries in the array).

I've got a fix for the problem, I need to pull out the debugging code I added to get to the bottom of it.

Issue occurs when we get /buttons/glide/{book id} and can't find the book ASIN in there - in _get_book_info_from_tooltips we'll skip over the book if this fails. For all books I've tried I never see ASIN in there, not sure when it would be there.

The fix is to grab the data from the information we already have, which has the bonus of reducing the number of URL fetches by (1 + number of other books by author).

szarroug3 commented 8 years ago

Nice catch. I think it only misses the information when there are multiple authors. All my books that have one author have normal information.

Which books are not working for the /buttons/glide part? I've had only one or two book not work there. Also, where are you going to get the ASIN for the "other books"?

stoduk commented 8 years ago

I've not found any book where /buttons/glide/{id} returns JSON containing the ASIN - either I'm picking bad books, or there is something else happening (I've tried it in Chrome, but maybe it is still some browser detection behaviour?).

Here are some that haven't worked: 29236280 29067264 23627014

Can you send the ID of a book that this worked for? Would be useful to see what the output looks like.

The ASIN comes from here (in _get_book_info_from_tooltips):

                asin_elements = book_data.xpath('//a[contains(@class, "kindlePreviewButtonIcon")]/@href')
                book_asin = urlparse.parse_qs(urlparse.urlsplit(asin_elements[0]).query)["asin"][0]
szarroug3 commented 8 years ago

That's super stranger, all three of those work for me 29067264 29236280 23627014

Also, the kindlePreviewButtonIcon thing only works on the page of the book you're trying to get the ASIN for and only if there is a preview of the book. At least that's what I can see.

stoduk commented 8 years ago

Mystery solved: it is a location thing.

From uk I get: {"glide_eligible":false,"glide_enabled":false,"ku_eligible":false,"marketplace_suffix":"_uk-21","ref_tag_override":null}

From uk but with VPN to USA: {"asin":"B01BMK1NBQ","glide_eligible":true,"glide_enabled":false,"ku_eligible":false,"ebook_price":null,"marketplace_suffix":"-20","ref_tag_override":null}

As for your other question, I'll take a look tomorrow, sleep is calling :)

szarroug3 commented 8 years ago

Makes sense.. I wonder if there's a way around that

On Mon, Oct 31, 2016, 6:55 PM Anthony Toole notifications@github.com wrote:

Mystery solved: it is a location thing.

From uk I get:

{"glide_eligible":false,"glide_enabled":false,"ku_eligible":false,"marketplace_suffix":"_uk-21","ref_tag_override":null}

From uk but with VPN to USA:

{"asin":"B01BMK1NBQ","glide_eligible":true,"glide_enabled":false,"ku_eligible":false,"ebook_price":null,"marketplace_suffix":"-20","ref_tag_override":null}

As for your other question, I'll take a look tomorrow, sleep is calling :)

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/szarroug3/X-Ray_Calibre_Plugin/issues/73#issuecomment-257454185, or mute the thread https://github.com/notifications/unsubscribe-auth/AIEjUaO-f0tteaNFq2jY6q8rVyxEEAcsks5q5n_jgaJpZM4Klc7D .

szarroug3 commented 8 years ago

Oh silly me! I misread your suggestion about the tooltips..Yeah that works!

On Mon, Oct 31, 2016, 6:58 PM Samreen Zarroug szarroug3@gmail.com wrote:

Makes sense.. I wonder if there's a way around that

On Mon, Oct 31, 2016, 6:55 PM Anthony Toole notifications@github.com wrote:

Mystery solved: it is a location thing.

From uk I get:

{"glide_eligible":false,"glide_enabled":false,"ku_eligible":false,"marketplace_suffix":"_uk-21","ref_tag_override":null}

From uk but with VPN to USA:

{"asin":"B01BMK1NBQ","glide_eligible":true,"glide_enabled":false,"ku_eligible":false,"ebook_price":null,"marketplace_suffix":"-20","ref_tag_override":null}

As for your other question, I'll take a look tomorrow, sleep is calling :)

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/szarroug3/X-Ray_Calibre_Plugin/issues/73#issuecomment-257454185, or mute the thread https://github.com/notifications/unsubscribe-auth/AIEjUaO-f0tteaNFq2jY6q8rVyxEEAcsks5q5n_jgaJpZM4Klc7D .

stoduk commented 8 years ago

Yes, the ASIN should be availalbe in the tooltips - for mainstream books it is always there, the only time it fails to find something is when looking up the "customer recommendations". These look to be more obscure books, so maybe not surprising they aren't on kindle..

See logs below - we now aren't getting any /buttons/glide/{id} resources as part of handling the main book, we are only doing it for the recommendations. We are doing an unnecessary lookup of the main title when handling duplicate author, but when I get around to adding caching that won't matter at all, so probably not worth thinking about.

FETCHING: /book/show/23627014-a-christmas-carol?from_search=true&search_version=service
FETCHING: /characters/395-timothy-cratchit
FETCHING: /characters/396-bob-cratchit
FETCHING: /characters/79460-jacob-marley
FETCHING: /characters/79532-ebenezer-scrooge
FETCHING: /work/quotes/3097440-a-christmas-carol
FETCHING: /author/show/239579.Charles_Dickens
FETCHING: /tooltips?resources[Book.1953][type]=Book&resources[Book.1953][id]=1953&resources[Book.2623][type]=Book&resources[Book.2623][id]=2623&resources[Book.5326][type]=Book&resources[Book.5326][id]=5326&resources[Book.18254][type]=Book&resources[Book.18254][id]=18254&resources[Book.58696][type]=Book&resources[Book.58696][id]=58696&resources[Book.31242][type]=Book&resources[Book.31242][id]=31242&resources[Book.169848][type]=Book&resources[Book.169848][id]=169848&resources[Book.31250][type]=Book&resources[Book.31250][id]=31250&resources[Book.5338][type]=Book&resources[Book.5338][id]=5338&resources[Book.5344][type]=Book&resources[Book.5344][id]=5344

@@@ we are looking up the main book again here, not too bothered

FETCHING: /author/show/69163.Arthur_Rackham
FETCHING: /tooltips?resources[Book.23627014][type]=Book&resources[Book.23627014][id]=23627014

@@@ looking up a lot of books here, all customer recommendations.  Most aren't on kindle so we are doing the fallback lookup (which fails from UK, may work from USA - from a check of one title it doesn't work, not a definitive test!)

FETCHING: /tooltips?resources[Book.563820][type]=Book&resources[Book.563820][id]=563820&resources[Book.1180333][type]=Book&resources[Book.1180333][id]=1180333&resources[Book.93531][type]=Book&resources[Book.93531][id]=93531&resources[Book.9993477][type]=Book&resources[Book.9993477][id]=9993477&resources[Book.42288][type]=Book&resources[Book.42288][id]=42288&resources[Book.20564][type]=Book&resources[Book.20564][id]=20564&resources[Book.78950][type]=Book&resources[Book.78950][id]=78950&resources[Book.1554931][type]=Book&resources[Book.1554931][id]=1554931&resources[Book.34431][type]=Book&resources[Book.34431][id]=34431&resources[Book.125321][type]=Book&resources[Book.125321][id]=125321&resources[Book.115538][type]=Book&resources[Book.115538][id]=115538&resources[Book.93818][type]=Book&resources[Book.93818][id]=93818&resources[Book.239810][type]=Book&resources[Book.239810][id]=239810&resources[Book.5319][type]=Book&resources[Book.5319][id]=5319&resources[Book.106749][type]=Book&resources[Book.106749][id]=106749&resources[Book.535856][type]=Book&resources[Book.535856][id]=535856&resources[Book.42036][type]=Book&resources[Book.42036][id]=42036&resources[Book.7331][type]=Book&resources[Book.7331][id]=7331&resources[Book.2452383][type]=Book&resources[Book.2452383][id]=2452383&resources[Book.549938][type]=Book&resources[Book.549938][id]=549938&resources[Book.3170878][type]=Book&resources[Book.3170878][id]=3170878&resources[Book.3027439][type]=Book&resources[Book.3027439][id]=3027439
FETCHING: /buttons/glide/93531
FETCHING: /buttons/glide/9993477
FETCHING: /buttons/glide/1554931
FETCHING: /buttons/glide/239810
FETCHING: /buttons/glide/42036
FETCHING: /buttons/glide/3027439
Job: 1 Creating Files finished
FETCHING: /book/show/29067264-harry-potter-and-the-cursed-child
FETCHING: /characters/28270-ron-weasley
FETCHING: /characters/82574-rose-weasley
FETCHING: /characters/956439-harry-potter
FETCHING: /characters/956440-hermione-granger
FETCHING: /characters/960125-albus-severus-potter
FETCHING: /characters/968685-scorpius-malfoy
FETCHING: /characters/968764-draco-malfoy
FETCHING: /work/quotes/48765776-harry-potter-and-the-cursed-child
FETCHING: /author/show/1077326.J_K_Rowling
FETCHING: /tooltips?resources[Book.3][type]=Book&resources[Book.3][id]=3&resources[Book.5][type]=Book&resources[Book.5][id]=5&resources[Book.136251][type]=Book&resources[Book.136251][id]=136251&resources[Book.6][type]=Book&resources[Book.6][id]=6&resources[Book.2][type]=Book&resources[Book.2][id]=2&resources[Book.15881][type]=Book&resources[Book.15881][id]=15881&resources[Book.1][type]=Book&resources[Book.1][id]=1&resources[Book.3950967][type]=Book&resources[Book.3950967][id]=3950967&resources[Book.862041][type]=Book&resources[Book.862041][id]=862041&resources[Book.13497818][type]=Book&resources[Book.13497818][id]=13497818
FETCHING: /author/show/3439408.Jack_Thorne
FETCHING: /author/show/5042201.John_Tiffany
FETCHING: /tooltips?resources[Book.29067264][type]=Book&resources[Book.29067264][id]=29067264
FETCHING: /tooltips?resources[Book.25446343][type]=Book&resources[Book.25446343][id]=25446343&resources[Book.28421168][type]=Book&resources[Book.28421168][id]=28421168&resources[Book.22359662][type]=Book&resources[Book.22359662][id]=22359662&resources[Book.26856502][type]=Book&resources[Book.26856502][id]=26856502&resources[Book.28170940][type]=Book&resources[Book.28170940][id]=28170940&resources[Book.26218178][type]=Book&resources[Book.26218178][id]=26218178&resources[Book.29385546][type]=Book&resources[Book.29385546][id]=29385546&resources[Book.30653853][type]=Book&resources[Book.30653853][id]=30653853&resources[Book.22299763][type]=Book&resources[Book.22299763][id]=22299763&resources[Book.22062213][type]=Book&resources[Book.22062213][id]=22062213&resources[Book.23566896][type]=Book&resources[Book.23566896][id]=23566896&resources[Book.28448239][type]=Book&resources[Book.28448239][id]=28448239&resources[Book.29384742][type]=Book&resources[Book.29384742][id]=29384742&resources[Book.20799109][type]=Book&resources[Book.20799109][id]=20799109&resources[Book.29358487][type]=Book&resources[Book.29358487][id]=29358487&resources[Book.28129423][type]=Book&resources[Book.28129423][id]=28129423&resources[Book.30687916][type]=Book&resources[Book.30687916][id]=30687916&resources[Book.29236299][type]=Book&resources[Book.29236299][id]=29236299&resources[Book.3][type]=Book&resources[Book.3][id]=3&resources[Book.15881][type]=Book&resources[Book.15881][id]=15881&resources[Book.5][type]=Book&resources[Book.5][id]=5&resources[Book.6][type]=Book&resources[Book.6][id]=6&resources[Book.2][type]=Book&resources[Book.2][id]=2&resources[Book.1][type]=Book&resources[Book.1][id]=1&resources[Book.136251][type]=Book&resources[Book.136251][id]=136251
FETCHING: /buttons/glide/25446343
FETCHING: /buttons/glide/28421168
FETCHING: /buttons/glide/22359662
FETCHING: /buttons/glide/26856502
FETCHING: /buttons/glide/28170940
FETCHING: /buttons/glide/26218178
FETCHING: /buttons/glide/29385546
FETCHING: /buttons/glide/30653853
FETCHING: /buttons/glide/22062213
FETCHING: /buttons/glide/23566896
FETCHING: /buttons/glide/28448239
FETCHING: /buttons/glide/29384742
FETCHING: /buttons/glide/20799109
FETCHING: /buttons/glide/29358487
FETCHING: /buttons/glide/28129423
FETCHING: /buttons/glide/30687916
FETCHING: /buttons/glide/29236299
Job: 2 Creating Files finished