MXD incorrectly identifies some books as missing from calibre

[WyndhamMisio]

The attached screenshot shows several eBooks as missing from calibre.

The Cassandra Clare books are in a different calibre library so that is correct. The others however, do exist in the current calibre library.

Or have I misunderstood something?

---

ps Very best wishes for a great 2014. :)

[uten]

That was for editing.

The error is for updated fanfics, using the fanfic downloader.

Using say "Harry Crow", when there is a new chapter, and you use the fanfic downloader to download it, that is when that error occurs. I do believe "Harry Crow" might work as I updated it yesterday. My current version has 86 chapters.

Your best bet is to find a prolific author, then download a few fanfics that are ongoing using the fanfic downloader plugin.. Then, update them when there is a new chapter. That should replicate the error.

Update an existing fanfic using the fanfic downloader plugin. The fanfic must be in calibre and in Marvin.

DO NOT transfer the updated fanfic.

Open MXD. Book will be Yellow.This is Wrong. It should be Gray.

Choose "Update Metadata to Marvin". Book will turn Green.This is the second error.

Close MXD.

Re-open MXD. Book will now be Gray.

Close MXD.

Transfer Book to Marvin.

Open MXD. Book will now be Green.

The first error is it reading as Yellow, when the metadata is correct, but the book contents have changed. The second error is a bad check caused by the first. Book turns Green when the metadata which is already identical is updated. It should be gray and stay gray.

The second error definitely seems to be caused by the first error, as it is doing what is supposed to as the book reads yellow in error. So fixing the first error should eliminate the second.

The most prolific author I know of is megamatt09, as s/he tends to update daily.

http://www.hpfanficarchive.com/stories/viewuser.php?uid=4810

Hope that helps.

[GRiker]

I've reimplemented the color logic. I'm not sure whether this will resolve the fanfic issues, but please install the latest revision of MXD to see if that makes a difference.

I have some theories about why fanfic updates might not be recognized, but I'd like to have you test this version before exploring those.

[GRiker]

Also, if you're still seeing the error it would be very helpful if you could create a debug log capturing it.

[uten]

No problem. I'll let you know as soon as I can test it. I have to wait for a fanfic to update. ;)

[GRiker]

@uten, I had an insight about why the updated fanfics were improperly identified as yellow instead of gray. I've updated both MXD and iOSRA, you will need to download and install both:

• Download iOSRA
• Download MXD

It's worth noting that an updated fanfic (or an edited epub) should never show yellow, only gray. If, after downloading a fanfic update a book is displayed as yellow, don't do the metadata update, but instead capture a debug log and post it.

Please let me know if this works properly.

[uten]

Tested it twice so far..

The book shows up as Gray. I then transferred the book to Marvin, and it showed up as green. The big problem is that every other book was gray.

I had to delete hashes and rebuild them.

Do you want a debug log of this?

[GRiker]

Yes, please send a debug log.

[GRiker]

Also, please clarify… Are you saying that after you rebuilt the hashes, everything worked properly? If everything is working properly now, a debug log won't tell me anything. However, if books are consistently being shown in gray that should be green, yes, I need to see a debug log of that.

[uten]

While everything is working right now - after rebuilding the hashes - I have made sure I left some fanfics to update so I can get the error to replicate. I'll send you the debug log in a few minutes.

[GRiker]

Another silly typo. Please do the following:

• Download and install updated MXD from this link
• After launching calibre, clear Marvin hashes (this should be the last time you need to do that)
• Launch MXD, see if things are working correctly.

I apologize, I should have caught that one but didn't. I really appreciate your assistance and persistence.

[uten]

Done. And done.. Works perfectly now.

And I was glad to help. :)

I will let you know if anything does crop up, but it looks awesome now.

[GRiker]

Whew. I will close this issue, feel free to reopen it. I'm planning to release this version after revising the help file to describe the new colors.

[GRiker]

@uten, I'd like your input on a further refinement please.

I'm planning to display a dialog when gray books are detected, just before showing the MXD window. This would be similar to the dialog displayed when duplicate content is detected. Additionally, I'm planning to mark duplicates and updated books with a temporary marker in the main calibre window to make it easier to isolate the books needing further attention.

An updated version of MXD with this feature is available for download from this link.

Here's how this will appear to the user:

• If there is duplicate content detected (magenta), the dialog will be shown (as before) with some additional text informing the user that the books are also marked with a temporary marker pin.
• If there is updated content detected (gray), a similar dialog will be shown with a list of books with updated content. These books will also be marked with a temporary marker pin.

The most likely scenario is that either one condition or the other is active - in other words, there is some duplicated content, or some updated content, but not both. In the event that both duplicated content and updated content are detected, all books would be marked with the temporary marker.

Questions:

• Do you like this feature?
• Do you think it's confusing to inform the user of both conditions at the same time when that occurs?
• This feature currently would delete any other temporary markers added by the user. Do you think that's a problem?

[uten]

You read my mind..

I was trying to figure out a way you could output a list of duplicates for the magenta books as just knowing I had them did make them hard to track down, but I had completely forgotten about temp markers. I'll create some magenta books, and some gray books, so I can see what the output looks like.

As for the third question, I think as long as they know they might lose their current temporary markers, there shouldn't be a problem - although given human nature, I'm sure someone will complain.. Regardless, both these conditions can't be found or fixed any other way, and help streamline both calibre's and Marvin's library. Lord knows enough of calibre's features already change the temp markers around. :)

[uten]

Ok. Got to have a good look at it and I think it's awesome!!

I didn't find it confusing at all, for multiple reasons. Firstly, you can copy the info, so you can refer to it if you want, and secondly, and most importantly, the information is in two separate windows, that appear one after the other, so the information isn't really given to you at the same time anyway. If it was all on the one screen, it may have been a little more confusing, depending on layout of course, but this isn't confusing at all.

I like that temp markers are added to all the books in both categories. The only thing I can think of to make everyone happy (which would be a miracle), is to make the temp marker selecting optional, which may not be possible. IE, add a tick box or something that lets them choose to select the Magenta books, the Gray Books, or both, or neither.. Hope that made sense, but it is probably far too much work to do that for the one or two grumpy people. :)

During one of my debug saving sessions, this error popped up. It happened either when I closed the MXD window, or when I quit calibre - I did both a little too close together to know for certain which. And it didn't happen the next time.

calibre, version 1.20.0 ERROR: Unhandled exception: AttributeError:'NoneType' object has no attribute 'stop'

calibre 1.20 isfrozen: True is64bit: True Darwin-13.0.0-x86_64-i386-64bit Darwin ('64bit', '') ('Darwin', '13.0.0', 'Darwin Kernel Version 13.0.0: Thu Sep 19 22:22:27 PDT 2013; root:xnu-2422.1.72~6/RELEASE_X86_64') Python 2.7.5 OSX: ('10.9.1', ('', '', ''), 'x86_64') Traceback (most recent call last): File "calibre_plugins.marvin_manager.action", line 405, in main_menu_button_clicked File "calibre_plugins.marvin_manager.action", line 827, in show_installed_books File "calibre_plugins.marvin_manager.action", line 363, in launch_library_scanner File "calibre_plugins.marvin_manager.action", line 390, in library_index_complete File "calibre_plugins.marvin_manager.action", line 871, in _busy_panel_teardown AttributeError: 'NoneType' object has no attribute 'stop'

[GRiker]

I've uploaded a new build of MXD at this link.

This version includes a checkbox in the config dialog General options section to disable the temp markers.

I also added some code to recover from the error you reported above.

[uten]

Very nice.

Only other suggestion is to split the checkbox into two - one for Magenta, and one for Gray - to let people work on just one group at a time if they want.

[GRiker]

Yes, definitely the right way to go. Download and install the latest version from this link.

Feel free to suggest better wording for the dialogs, I know it's awkward. Note that the dialog text changes depending on whether flagging is enabled for each condition.

[uten]

Error found in the new selection.

If both options are ticked, only the updated content gets temporary markers. If they are selected independently (only one at a time), they work as they should.

As for the dialogs, I don't think they are awkward at all. They state the conditions and info needed, without any extraneous, confusing text added. You also have separated the points by paragraph breaks instead of bunching the information all together. I think the dialogs are well done.

[GRiker]

You are good tester. I thought I had tested for that.

Updated version available from this link. A few more tweaks to the text in both the dialog tooltips and the dialogs.

[uten]

Very nice. Works perfectly now..

I've been thinking, and we have been using the term updated. We should have been using differing, as it is possible to add the book to Marvin, then transfer it to calibre. And it shows as gray, which it should.

So instead of updated content, maybe call it differing content.

What really called it out to me, was you use the sentence "Marvin books with outdated content" in the dialog window, but it's not necessarily true.

As an example - I could use an online downloading service on my idevice to get an updated novel/fanfic/magazine whatever, then save it to Marvin. That puts the new version in Marvin, and the old one in calibre - and is a lot more work given that you have to also delete the old copy from calibre and you lose any metadata edits.... But you see what I mean???

[GRiker]

You're right, that path is perfectly valid. I will think about the wording, and also how to possibly handle a backload from Marvin to calibre without losing metadata.

[GRiker]

@uten, here's a lengthy brain dump of the situation as I see it.

First, here's a summary of how content mismatches (displayed as GRAY) are detected:

• When MXD is launched, it generates a variety of information about books in the calibre library and books in the Marvin library.
• One of the data points is calibre's opinion of whether or not a book is installed on the device, indicated by a checkmark in the On Device column in the Library window, and the In Library column in the Device window. Calibre displays a checkmark in those columns if either a) the UUIDs in the books match, or b) the title/author match.
• If calibre thinks a book is installed in Marvin, the color selection algorithm will assign GREEN, YELLOW, ORANGE, MAGENTA or GRAY. GRAY is currently assigned when no other color can be assigned.
• If calibre thinks a book is not installed in Marvin, the color selection algorithm assigns WHITE or RED.

Now, let's consider typical workflows for adding books from calibre to Marvin, then tweaking in calibre:

• User adds a book to calibre. Calibre assigns a UUID to the book.
• User sends that book to Marvin. The book is shown with a checkmark in both the Library and Device windows.
• User launches MXD. The book is shown in GREEN, meaning that the book is in both calibre and Marvin, the UUID and metadata match, with a single copy in each location.
• User closes MXD, and edits metadata in calibre or Marvin, then relaunches MXD. Book is shown in YELLOW, indicating a metadata mismatch. User has option of resolving, using either source as the master.
• User downloads an updated version of the book (your fanfic scenario) or edits the book from calibre. The UUID doesn't change, as it's still the same entry in the calibre library, but the content has changed, causing a hash mismatch. Book is shown in GRAY.

Here's the workflow when adding a book from Marvin to calibre:

• User adds a book from some external source to Marvin.
• User launches MXD. The book is shown in WHITE, as it's not matched with any book in the calibre library.
• User selects Add to calibre library from the context menu in MXD.
• MXD adds the book to calibre's library. Marvin is informed of the calibre-assigned UUID, which will enable the book to be displayed as GREEN in future MXD sessions.

We've already seen that if a user-initiated change occurs to a book in the calibre library, say downloading an updated fanfic or editing the book, the UUID doesn't change, allowing MXD to detect that the book's identity is the same, but the content has been updated, resulting in GRAY.

But what happens when the user replaces a book in Marvin? (Users cannot directly edit a book's content in Marvin, only some of the metadata.) If the user tries to re-download a book whose author and title match a book already installed in Marvin, Marvin does not replace it. This is a safety feature, as Marvin retains lots of state information about a book, including last read position, annotations, bookmarks, etc, all of which would be invalidated by replacing the underlying content of the book. So the only way to truly replace a book in Marvin, using Marvin's UI, is to a) delete the book from Marvin's library, then b) re-download it. Doing so creates an new entry in Marvin's database, without a matching calibre UUID.

This suggests a particular scenario in which it might be possible to fool the current color algorithm:

• User adds a book from an online source to Marvin
• User transfers that book from Marvin to calibre using MXD's Add to calibre library feature
• User deletes that book in Marvin, and re-downloads a refreshed version from the online source
• User launches MXD. Calibre says it recognizes the book, as the Title and Author match. MXD assigns GRAY, because the hashes don't match.
• How would the user get the latest copy of the book from Marvin's library to calibre? The best way (currently) would be to delete the book from the calibre library, and re-add it to calibre using the Add to calibre library feature, which would get the UUIDs back in sync. This has the undesired side effect of losing any metadata edits the user may have made on the calibre side.

My sense is that this is an unlikely scenario - once a Marvin user starts using calibre to manage content, they would tend to do all content management from calibre, not Marvin. The recommended approach would be to add the latest version of the book to the existing entry in calibre, retaining metadata and UUID.

If I were to make a change, here's how I think it would work:

• Display the book in a new color, TEAL, meaning that there is a title/author match between libraries, but a content and UUID mismatch - in other words, the book was updated in Marvin. GRAY would mean the UUIDs match, but the content does not. I would change the terminology for GRAY to 'content mismatch', to make it similar to YELLOW's 'metadata mismatch'. TEAL would mean 'updated content in Marvin'.

The implementation would be a context menu item for TEAL books, Update in calibre library, which would fetch a copy of the Marvin version of the book, apply that copy to the existing calibre entry, then update Marvin's UUID for that book to match the already-existing calibre entry for that title/author. This would retain existing metadata in calibre and get everything back in sync.

I need to do more research to confirm that this is possible. It would be challenging to test.

---

So, in summary, your scenario is certainly possible, but an informed user could avoid the problem by adding the updated content to calibre rather than Marvin. A more elegant solution would be to add a new color TEAL representing updated Marvin content, with a context menu action updating the calibre library from the Marvin book.

[GRiker]

I've updated the plugin to revise the terminology from 'outdated content' to 'differing content'. You can download the latest version of the plugin from this link.

[uten]

Wow. Lots of info to think about.

I happen to agree that this is an unlikely situation. I only discovered it because I was actively attempting to check everything. I personally would never use Marvin in that way, unless I had to travel for a while. And that point there was why I brought it up.

I do like your solution, both simple and elegant.

Once you determine whether or not it's even possible, maybe a poll to see how many people would like the Teal color. Until then, a simple note, letting them know how to avoid the 'problem', would work.

I'll keep thinking on it, and if I think of anything else, I'll let you know.

[GRiker]

With respect to the colors, what I've been doing is subdividing the color wheel, trying to pick colors as far apart as possible so they can be distinguished by folks with vision problems. So it's not personal taste, it's maximizing visible difference.

I picked GRAY because it suggested something broken. One option with GRAY would be to do a lighter shade and a darker shade, which would keep them grouped and visibly related. But first I need to confirm the other requirements before fine-tuning the color assignment.

I do appreciate the time and effort you've invested to help improve MXD - I wish you had been around when I was originally developing it. The recent improvements are significant.

[GRiker]

Out of curiosity, I added the code to subdivide GRAY into two groups - those where the content has been updated in calibre, and those where the content has been updated in Marvin. Calibre updates are still shown as GRAY, Marvin updates are shown as CYAN. You can download it from this link. To clarify, this is the only change - the CYAN group is treated identically to the GRAY group.

I hacked Marvin's database to force a CYAN book in my test library for testing purposes, but I'm curious to know if any of your books show up as CYAN?

[uten]

I added a few books and edited a few more to try and get all the colors to show.

I had all the colors show up. Cyan included.

On a side note. There may be a problem with using light gray to do with viewing angle and color profiles... My three year old knocked my monitor askew while I was fixing up the library, and I went from having a white book and two gray books to having three off-white books. I took a screenshot (attached below) and checked on a windows desktop, a windows laptop and an ipad, and the only one that displayed it correctly at any angle was my ipad. Both windows computers displayed the gray lighter than the Mac, but the worst was the laptop despite the fact I got it looking identical to the ipad at the correct viewing angle.

And even minor messing with the color profile also turned all three off-white..

Not sure if it's worth worrying about too much, but I also know there aren't many monitors that have the correct color profile. And it was only white and light gray that were affected like this.

On the upside, the white and light gray were able to help me fine tune my color profiles so they showed correctly on my Windows computers.

I couldn't decide if this was nitpicking or not, so I figured I'd let you decide. :)

[GRiker]

I darkened GRAY by 5%, that should help. Download from this link.

[uten]

That did it. You can still tell the difference, no matter what now. :)

[GRiker]

I've made another revision which you can download from this link. This version shows books updated in calibre with a light gray background, and books updated in Marvin with a dark gray background. Books updated in either location will be reported together in the Updated content dialog box. The details section lists the books separately.

I'm still thinking about whether or not to add the functionality to update books from within the MXD dialog. If I do this, it will need to be bidirectional - in other words, you should be able to send a book updated in calibre to Marvin, and a book updated in Marvin should update the calibre copy.

For the moment, the books are identified with enough details that the user can resolve the issues manually.

[uten]

Very nice. There is certainly enough information to resolve things quite easily.

The problem with updating books via MXD isn't just that. Sometimes, the updated book will alter the metadata (especially true in Marvin as it creates a new book - in calibre you can usually disable it). So not only do you need to transfer the books in both directions, the option would need to be there to transfer the 'old' metadata to the updated book, so that any edits someone made aren't lost.

Resolving the issue manually make take a little longer, but it isn't difficult at all.