Import failed but not sure why

[dkids]

I've replaced the username and password. Import does seem to work almost all of the time. If this is just a temporary problem it would be nice if it re-tried X times.
hunted.txt

moth.txt

Both books are in Calibre.

[philborman]

We seem to be getting incomplete responses from calibredb, or somehow truncating the response, so we miss the "bookid" that calibre sends us back. Both books show calibredb returned 0 which means success, but the res[ ] message which should contain the bookid (among other things) is incomplete. Not sure why at this stage, need to think about it...

Can you check your logs for a book that did complete successfully, should see the whole res [ ] message

[dkids]

side.txt summer.txt Here are two.

[philborman]

ok thanks. The results in the log are still incomplete, but not sure why.
We are expecting calibredb returned 0: res [ some text ] err [ some text or blank ] but we don't get the closing ] on res, or any err[ ] message.

Both books that succeeded show calibredb returned 0: res[Title : Cold Summer and both books that failed show calibredb returned 0: res[ /Users/kids/Downloads/Books

Looks as though the ones that succeeded have more in res[ ] than the log shows, as we look for the bookid in res, and complain if we don't find it. Need to figure out why the log line is incomplete so we can see what's happening.

[philborman]

New version of lazylibrarian has more logging in the calibre communication section, hopefully will show what's going on...

[dkids]

Thanks! I'll post them as I see them. First one, which looks like it should not have matched and should have been imported:

08-May-2018 13:52:54 - DEBUG :: POSTPROCESS : calibre.py:calibredb:433 : calibredb rc 0 08-May-2018 13:52:54 - DEBUG :: POSTPROCESS : calibre.py:calibredb:440 : calibredb res 148 [ /Users/kids/Downloads/Books/Ann Brashares - Sisterhood of the Traveling Pants Series/Th e Sisterhood of the Traveling Pants - Ann Brashares.epub ] 08-May-2018 13:52:54 - DEBUG :: POSTPROCESS : calibre.py:calibredb:441 : calibredb err 202 8[The following books were not added as they already exist in the database (see --duplicates option): Title : Sisterhood Everlasting Title sort : Sisterhood Ev erlasting Author(s) : Ann Brashares [Brashares, Ann] Publisher : Random House Tags : Fiction, General, Contemporary Women, Female Friendship Language s : eng Rating : 4.5 Published : 2011-06-01T00:00:00+00:00 Rights : Copyright © 2011 by Ann Brashares Identifiers : isbn:978161707 5742 Comments :

Return to the Sisterhood of the Traveling Pants . . . ten years later
 
From #1 New York Times bestselling au thor Ann Brashares comes the welcome return of the characters whose friendship became a touc hstone for a generation. Now Tibby, Lena, Carmen, and Bridget have grown up, starting their lives on their own. And though the jeans they shared are long gone, the sisterhood is everla sting.

Despite having jobs and men that they love, each knows that something is missin g: the closeness that once sustained them. Carmen is a successful actress in New York, engag ed to be married, but misses her friends. Lena finds solace in her art, teaching in Rhode Is land, but still thinks of Kostos and the road she didn’t take. Bridget lives with her longti me boyfriend, Eric, in San Francisco, and though a part of her wants to settle down, a bigge r part can’t seem to shed her old restlessness.

Then Tibby reaches out to bridge the d istance, sending the others plane tickets for a reunion that they all breathlessly await. An d indeed, it will change their lives forever—but in ways that none of them could ever have e xpected.

As moving and life-changing as an encounter with long-lost best friends, Sisterhood Everlasting is a powerful story about growing up, losing your way, and findi ng the courage to create a new one.

Formats : EPUB, MOBI Quality
: 5 ] 08-May-2018 13:52:54 - ERROR :: POSTPROCESS : postprocess.py:processDir:800 : Postprocessi ng for The Sisterhood of the Traveling Pants - Ann Brashares has failed: Calibre failed to i mport Ann Brashares The Sisterhood of the Traveling Pants, no added bookids 08-May-2018 13:52:54 - WARNING :: POSTPROCESS : postprocess.py:processDir:821 : Residual files remain in /Users/kids/Downloads/Books/Ann Brashares - Sisterhood of the Traveling Pants Series.fail

[dkids]

I'm assuming the error message is related to the import of the one book. The book is listed as "snatched" rather than "open", as were 3 other books by this author that were successfully sent to Transmission and imported.

08-May-2018 13:52:53 - DEBUG :: POSTPROCESS : librarysync.py:LibraryScan:443 : [/Users/kid s/Calibre/Elise Kova/] Now scanning subdirectory Water's Wrath (Air Awakens #4) (7833) 08-May-2018 13:52:53 - DEBUG :: POSTPROCESS : librarysync.py:LibraryScan:477 : book meta [ ] [en] [Elise Kova] [Water's Wrath (Air Awakens Series Book 4)] [epub] 08-May-2018 13:52:53 - DEBUG :: POSTPROCESS : librarysync.py:find_book_in_db:170 : Searchi ng database for [Waters Wrath (Air Awakens Series Book 4)] by [Elise Kova] 08-May-2018 13:52:53 - DEBUG :: POSTPROCESS : librarysync.py:find_book_in_db:206 : Searchi ng 19 books by [Elise Kova] in database for [Waters Wrath (Air Awakens Series Book 4)] 08-May-2018 13:52:53 - DEBUG :: POSTPROCESS : librarysync.py:find_book_in_db:253 : Fuzz ma tch partial [95] [Waters Wrath (Air Awakens Series Book 4)] [Air Awakens] 08-May-2018 13:52:53 - DEBUG :: POSTPROCESS : librarysync.py:LibraryScan:794 : Librarysync link to preferred type epub: /Users/kids/Calibre/Elise Kova/Water's Wrath (Air Awakens #4) (7833)/Water's Wrath (Air Awakens #4) - Elise Kova.epub 08-May-2018 13:52:53 - WARNING :: POSTPROCESS : librarysync.py:LibraryScan:808 : Updating bo ok location for Elise Kova Waters Wrath (Air Awakens Series Book 4) from /Users/kids/Calibre /Elise Kova/Air Awakens (7831)/Air Awakens - Elise Kova.epub to /Users/kids/Calibre/Elise Ko va/Water's Wrath (Air Awakens #4) (7833)/Water's Wrath (Air Awakens #4) - Elise Kova.epub 08-May-2018 13:52:53 - DEBUG :: POSTPROCESS : librarysync.py:LibraryScan:811 : Elise Kova Waters Wrath (Air Awakens Series Book 4) matched Open BookID 23127048, [Elise Kova][Air Awak ens] 08-May-2018 13:52:53 - INFO :: POSTPROCESS : librarysync.py:LibraryScan:883 : 1/1 new/mod ified eBooks found and added to the database 08-May-2018 13:52:53 - INFO :: POSTPROCESS : librarysync.py:LibraryScan:884 : 14 files pr ocessed 08-May-2018 13:52:53 - DEBUG :: POSTPROCESS : librarysync.py:LibraryScan:979 : Updating bookcounts for 1 author 08-May-2018 13:52:53 - INFO :: POSTPROCESS : librarysync.py:LibraryScan:983 : Library scan complete 08-May-2018 13:52:53 - DEBUG :: POSTPROCESS : postprocess.py:processDir:698 : Processed torznab: Water's Wrath (Air Awakens #4) - Elise Kova, magnet:?xt=urn:btih:137d13808d7f2b298f2f374b3d892f272fdd6495&dn=Water's+Wrath+(Air+Awakens+Seri+-+Elise+Kova.epub.torrent 08-May-2018 13:52:53 - ERROR :: POSTPROCESS : postprocess.py:processExtras:1252 : processExtras: No dest_file supplied 08-May-2018 13:52:53 - WARNING :: POSTPROCESS : postprocess.py:processDir:746 : TRANSMISSION is seeding torznab Water's Wrath (Air Awakens Seri - Elise Kova.epub 08-May-2018 13:52:53 - INFO :: POSTPROCESS : postprocess.py:processDir:786 : Successfully processed: Water's Wrath (Air Awakens #4) - Elise Kova

[philborman]

Calibre failed to import Ann Brashares The Sisterhood of the Traveling Pants, no added bookids calibre reported The following books were not added as they already exist in the database so it thinks the book is already there? Calibre uses the embedded book metadata, so maybe that's the problem?

Elise Kova Waters Wrath You seem to have two copies in the calibre database, bookid (7831) and (7833) Probably best to delete one of them, but check formats in case one has more than the other.

[dkids]

The books seem to have been imported properly. Here's the titles under the author:

ls Elise\ Kova/ Air Awakens (7831) The Alchemists of Loom (7618) Crystal Crowned (7832) Water's Wrath (7833) Earth's End (7830)

Here's the contents of the directories:

ls Elise\ Kova/* Elise Kova/Air Awakens (7831): Air Awakens - Elise Kova.epub metadata.opf cover.jpg

Elise Kova/Crystal Crowned (7832): Crystal Crowned - Elise Kova.epub metadata.opf cover.jpg

Elise Kova/Earth's End (7830): Earth's End - Elise Kova.epub metadata.opf cover.jpg

Elise Kova/The Alchemists of Loom (7618): The Alchemists of Loom - Elise Kova.mobi cover.jpg metadata.opf

Elise Kova/Water's Wrath (7833): Water's Wrath - Elise Kova.epub metadata.opf cover.jpg

Not sure what I'm looking for so I've attached "grep -i kova /Users/kids/src/LazyLibrarian-master/Logs/lazylibrarian.log". I'd be happy to do other searches.

kova.txt

The name of the series is "Air Awakens" and also the title of the first book (which seems like about 50% of the time the first book is the title for the series).

I did see the Error 1252 message on other books that were successfully imported.

08-May-2018 13:52:42 - DEBUG :: POSTPROCESS : postprocess.py:processDir:698 : Processed to rznab: Earth's End - Elise Kova, magnet:?xt=urn:btih:d91c0c0b08a4ab920747c8dda0c13a02b353f96 b&dn=Earth's+End+(Air+Awakens+Series+-+Kova%2C+Elise.epub.torrent 08-May-2018 13:52:42 - ERROR :: POSTPROCESS : postprocess.py:processExtras:1252 : processExtras: No dest_file supplied 08-May-2018 13:52:42 - WARNING :: POSTPROCESS : postprocess.py:processDir:746 : TRANSMISSION is seeding torznab Earth's End (Air Awakens Series - Kova, Elise.epub 08-May-2018 13:52:42 - INFO :: POSTPROCESS : postprocess.py:processDir:786 : Successfully processed: Earth's End - Elise Kova

08-May-2018 13:52:42 - DEBUG :: POSTPROCESS : postprocess.py:move_into_subdir:201 : copy_into_subdir Earth's End (Air Awakens Series - Kova, Elise.epub 08-May-2018 13:52:42 - DEBUG :: POSTPROCESS : postprocess.py:processDir:560 : Found folder (96%) [/Users/kids/Downloads/Books/Earth's End (Air Awakens Series - Kova, Elise.unpack] for eBook Elise Kova - Air Awakens (Air Awakens #1).epub

[philborman]

Ok, I think I see what's going on here. Earth's and Water's containing apostrophes are being compared to the words without the apostrophe Earths Waters and that's pulling down the fuzzy match percentage .

fuzz.token_sort_ratio("Waters Wrath", "Water's Wrath") = 88 which is below the threshold.

We need to strip the apostrophes before comparison so we get the right match. Can you run the same import with debug level set to 3, lazylibrarian will print each book it compares against and the percentage match. Should confirm if that's the problem. If it is, should be fairly simple to fix.

[dkids]

patchett.txt Book has been downloaded succesfully but LL is not trying to import it.

ls -ald Ann\ Patchett\ State\ of\ Wonder\ 2011\ Retail\ EPUB\ eBook-BitBook/ drwxr-xr-x@ 5 kids staff 170 May 10 13:11 Ann Patchett State of Wonder 2011 Retail EPUB eBook-BitBook/ ls -al Ann\ Patchett\ State\ of\ Wonder\ 2011\ Retail\ EPUB\ eBook-BitBook/ total 7280 drwxr-xr-x@ 5 kids staff 170 May 10 13:11 . drwxr-xr-x 22 kids staff 748 May 10 14:49 .. -rw-r--r-- 1 kids staff 3697839 May 10 13:11 bb-State_of_Wonder.epub -rw-r--r-- 1 kids staff 10865 May 10 13:11 bitbook.nfo -rw-r--r-- 1 kids staff 640 May 10 13:11 file_id.diz

The log files are huge. I am at level 3, as you asked, and I ran through 100 log files in about 8 hours last night. And the log files are 10MB each. I had RSS feeds set to 20 minutes which I changed to 0 minutes last night. That - to me - seems like it should be RSS off.

I'm not complaining, just having trouble giving you just the relevant info without a lot of other stuff thrown in. I've tried running "Run PostProcessor" but I get a "Skipping (not LL)" error:

10-May-2018 15:09:38 - DEBUG :: POSTPROCESS : postprocess.py:check_residual:922 : Skipping (not LL) Joe R Lansdale - Mucho Mojo 10-May-2018 15:09:38 - DEBUG :: POSTPROCESS : postprocess.py:check_residual:922 : Skipping (not LL) Ann Leckie - [Imperial Radch 03] - Ancillary Mercy (retail).epub 10-May-2018 15:09:38 - DEBUG :: POSTPROCESS : postprocess.py:check_residual:922 : Skipping (not LL) Ann Patchett State of Wonder 2011 Retail EPUB eBook-BitBook

Here are all the files in the directory:

Adrian Tchaikovsky - Children of Time 2015 (Sci-Fi) ePUB+MOBI Alexandra Christo - To Kill a Kingdom LL.(34499221).fail Ann Leckie - [Imperial Radch 03] - Ancillary Mercy (retail).epub Ann Patchett State of Wonder 2011 Retail EPUB eBook-BitBook Collins, Suzanne - [Distritos 02] Catching fire [5944] (r1.2 JerGeoKos).epub Collins, Suzanne - [Distritos 02] Catching fire [5944] (r1.2 JerGeoKos).unpack Crouch, Blake - [Wayward Pines 01] Pines [23489] (r1.0) [EN].epub Crouch, Blake - [Wayward Pines 01] Pines [23489] (r1.0) [EN].unpack Death Doesn't Bargain by Sherrilyn Kenyon EPUB Deaths End The Three-Body Problem - Cixin Liu Fangirl - Rainbow Rowell.epub Fangirl - Rainbow Rowell.unpack Joe R Lansdale - Mucho Mojo Laini Taylor - Night of Cake & Puppets (Daughter of Smoke and Bone #2.5) epub & mobi.fail Palacio R J - Wonder - Italian Edition-Bymonello78.pdf Rowell, Rainbow - Fangirl [20912] (r1.0 Cwpper).epub The Goldfinch-Donna Tartt BY ATHESSI13.zip The Traitor's Game by Jennifer A. Nielsen EPUB Torment

I'm not sure if the problem is that LL is trying to match on the file name, which is less relevant than the directory name (which has title and author both).

Anyway, I did this command in an effort to filter out duplicate searches, etc.
grep -i "Patchett" ~/src/LazyLibrarian-master/Logs/lazylibrarian.log* | cut -f10- -d: |sort -u > ~/patchett.txt

[dkids]

The two fails were because the book already existed by the time LL tried to import it.

calibredb err 364[The following books were not added as they already exist in the database (see --duplicates option): Title : To Kill a Kingdom Author(s) : Alexandra Christo [Alexandra Christo] Publisher : Hot Key Languages : eng Published : 2018-02-21T08:00:00+00:00 Identifiers : urnisbn/9781471707406:9781471707406 ] Postprocessing for To Kill a Kingdom - Alexandra Christo has failed: Calibre failed to import Alexandra Christo To Kill a Kingdom, no added bookids

Users/kids/src/LazyLibrarian-master/Logs/lazylibrarian.log.3:10-May-2018 14:19:51 - DEBUG :: POSTPROCESS : postprocess.py:processDestination:1342 : Importing Daughter of Smoke & Bone - Laini Taylor into calibre library /Users/kids/src/LazyLibrarian-master/Logs/lazylibrarian.log.3:10-May-2018 14:19:51 - DEBUG :: POSTPROCESS : calibre.py:calibredb:426 : [u'/Applications/calibre.app/Contents/console.app/Contents/MacOS/calibredb', 'add', '--username', u'dkids', '--password', u'DobyTang', '-1', '--with-library', u'http://127.0.0.1:8080/#Calibre', u'/Users/kids/Downloads/Books/Laini Taylor - Night of Cake & Puppets (Daughter of Smoke and Bone #2.5) epub & mobi'] /Users/kids/src/LazyLibrarian-master/Logs/lazylibrarian.log.3:10-May-2018 14:19:52 - DEBUG :: POSTPROCESS : calibre.py:calibredb:440 : calibredb res 485[ /Users/kids/Downloads/Books/Laini Taylor - Night of Cake & Puppets (Daughter of Smoke and Bone #2.5) epub & mobi/Daughter of Smoke & Bone - Laini Taylor.opf /Users/kids/Downloads/Books/Laini Taylor - Night of Cake & Puppets (Daughter of Smoke and Bone #2.5) epub & mobi/Daughter of Smoke & Bone - Laini Taylor.epub /Users/kids/Downloads/Books/Laini Taylor - Night of Cake & Puppets (Daughter of Smoke and Bone #2.5) epub & mobi/Daughter of Smoke & Bone - Laini Taylor.mobi ] /Users/kids/src/LazyLibrarian-master/Logs/lazylibrarian.log.3:10-May-2018 14:19:52 - DEBUG :: POSTPROCESS : calibre.py:calibredb:441 : calibredb err 1704[The following books were not added as they already exist in the database (see --duplicates option): Title : Night of Cake & Puppets Title sort : Night of Cake & Puppets Author(s) : Laini Taylor [Taylor, Laini] Publisher : Little, Brown Books for Young Readers Book Producer : calibre (1.11.0) [http://calibre-ebook.com] Series : Daughter of Smoke & Bone #2.50 Languages : eng Timestamp : 2013-11-27T01:33:19.835000+00:00 Published : 2013-11-25T14:00:00+00:00 Identifiers : isbn13:9780316369855 Comments :

In Night of Cake & Puppets, Taylor brings to life a night only hinted at in the Daughter of Smoke & Bone trilogy—the magical first date of fan-favorites Zuzana and Mik. Told in alternating perspectives, it’s the perfect love story for fans of the series and new readers alike. Petite though she may be, Zuzana is not known for timidity. Her best friend, Karou, calls her “rabid fairy,” her “voodoo eyes” are said to freeze blood, and even her older brother fears her wrath. But when it comes to the simple matter of talking to Mik, or “Violin Boy,” her courage deserts her. Now, enough is enough. Zuzana is determined to meet him, and she has a fistful of magic and a plan. It’s a wonderfully elaborate treasure hunt of a plan that will take Mik all over Prague on a cold winter’s night before finally leading him to the treasure: herself! Violin Boy’s not going to know what hit him.

] /Users/kids/src/LazyLibrarian-master/Logs/lazylibrarian.log.3:10-May-2018 14:19:52 - ERROR :: POSTPROCESS : postprocess.py:processDir:800 : Postprocessing for Daughter of Smoke & Bone - Laini Taylor has failed: Calibre failed to import Laini Taylor Daughter of Smoke & Bone, no added bookids

[philborman]

The log files are huge. I am at level 3, as you asked, and I ran through 100 log files in about 8 hours last night. And the log files are 10MB each. I had RSS feeds set to 20 minutes which I changed to 0 minutes last night. That - to me - seems like it should be RSS off.

Yes, could easily add that. At the moment you turn rss off by unticking the rss provider boxes.

I'm not complaining, just having trouble giving you just the relevant info without a lot of other stuff thrown in. I've tried running "Run PostProcessor" but I get a "Skipping (not LL)" error:

No problem, it's improving the program! Run Postprocessor checks for snatched books and then processes any that weren't snatched but match a LL naming pattern, this is to pick up wishlist books or books from csv lists. You can ignore that message, it's just for info, something was found in our download directory that we think isn't ours so we ignored it.

I might look into fine tuning the loglevels > 2 so we can set different levels to only give in-depth reports on small sections