GRiker
commented
10 years ago I'm not understanding the issue. Please be more specific which books are missing.
Closed WyndhamMisio closed 10 years ago
GRiker
commented
10 years ago I'm not understanding the issue. Please be more specific which books are missing.
WyndhamMisio
commented
10 years ago The books missing are shown in the MXD window with a white background, apart from the Cassandra Clare books.
ie: Black Wings, Blood Magic, Bloody Skies, Picked at the Peak, The Preachers's Son, When Extended Family is Discovered and, When I Saw You.
Some of these are shown in the attached screenshot which has the current calibre library showing in the background.
eg: Black Starr Cross, Blood Magic and, Bloody Skies. The others do exist in calibre but are not visible in the current screenshot.
The MXD window has nearly two thousand books with a white background and my assumption is that this is probably correct as I synchronize Marvin from two calibre libraries.
I only spotted these, because I sorted by reading percentage to find the most recently read books. Consequently I can't say if the books noted above are the only ones shown in error.
GRiker
commented
10 years ago Hi Wyndham, Happy new year.
If the MXD window shows a book in white, that means that the plugin did not find a matching book in your current calibre library.
In your screenshot, the most likely explanation for the books you're expecting to shown as matched is that the author doesn't match between calibre and Marvin, even though the title does match. For a match to be detected, both the author and title must match exactly.
Additionally, the content of the book needs to match. If you have a book in Marvin, and a different version of the same book in calibre, they won't match. This can often happen if you add books from different sources. The easiest way to detect this is to simply open the book in calibre's viewer, then open the same book in Marvin. Most likely you will see that they are different versions of the same book.
It's also possible that there's another problem with the plugin, but these are the most likely explanations. Please close this issue if that's what you find, or provide additional metadata details about the books shown incorrectly.
WyndhamMisio
commented
10 years ago I can confirm they have the same author. All the books in Marvin have been sourced from Calibre only, so it seems unlikely to me that the books are different.
If I update a book and then convert again, I usually download straight away to the mobile devices. In this case, I am fairly confident about these books as they are all recent reads.
Simple visual checks on the books show the following:
· Black Wings appears identical
· Blood Magic appears identical
· Bloody Skies appears identical
As a final test I sent these three books from Calibre to Marvin and re-ran MXD, and they no longer show as missing from Calibre. This leaves me a little frustrated.
I have to assume that it is either not possible, or easy, programmatically to distinguish between a book not existing (under the current rules, because the content is different in some, possibly small, respect) and to identify the book as possibly existing and the same book, but with differences. This could include slight differences in the title or author, such as Midnight Heir and The Midnight Heir, J.D. Robb and JD Robb, or Arthur Clarke and Arthur C. Clarke, etc.
There remain a few other books which are not being correctly identified (given the aforementioned provisions), so I can do further testing if you like.
uten
commented
10 years ago I have the same problem. I figured it might be a glitch from multiple versions of calibre/the plugins/Marvin, so I deleted every book from Marvin, Polished all the books to verify calibre and book metadata was identical and finally transferred them from Calibre.
Around half my Library of 500 books is 'not in calibre' even though I JUST copied it directly from calibre.
After some more experimenting, I also discovered when I select a single white colored book - in the Marvin Library subwindow of calibre, I don't even have the correct options if the books was missing (I only have View Metadata, and Delete Book).
However, if I select multiple books, I have the options to Delete Book, Export Metadata or Import Metadata. Even though the Library window says they are missing from calibre, I am not allowed to import the book to calibre.
Choosing 'Export Metadata from calibre' causes every selected white book to turn green. Once I close the Marvin Library and reopen it, they are white again. This only occurs with these mistakenly white books - I altered metadata in calibre to make a few books yellow and they updated properly and stayed green.
Hope that helps pinpoint the problem.
GRiker
commented
10 years ago If you can give me a specific set of steps that lead to the failure, along with a debug log, that would help pinpoint the problem.
GRiker
commented
10 years ago Here are a few more steps that would help in pinpointing the problem.
How is the book displayed in MXD?
If it's other than green, please post the debug log here in a new comment
uten
commented
10 years ago Ok, here you go. Did as you said. Book is still white.
Focused on a single book - Harry Crow by robst. I use Marvin for Fanfics mostly. Sorry the debug log is so long, but I still have a bunch of fanfic covers to make.
Also, I use the latest version of all apps/plugins and run calibre on OSX 10.9.1
calibre 1.19 isfrozen: True is64bit: True
Darwin-13.0.0-x86_64-i386-64bit Darwin ('64bit', '')
('Darwin', '13.0.0', 'Darwin Kernel Version 13.0.0: Thu Sep 19 22:22:27 PDT 2013; root:xnu-2422.1.72~6/RELEASE_X86_64')
Python 2.7.5
OSX: ('10.9.1', ('', '', ''), 'x86_64')
DEBUG: 0.0 iOSReaderApp:initialize(v1.3.4)
DEBUG: 0.0 iOSReaderApp:compile_ui()
DEBUG: 0.1 iOSReaderApp:_init_prefs(prefs created under v1.3.0)
DEBUG: 0.1 debug_libimobiledevice: False
DEBUG: 0.1 debug_plugin: True
DEBUG: 0.1 development_mode: False
DEBUG: 0.1 kindle_enabled_formats: [u'MOBI']
DEBUG: 0.1 kindle_supported_formats: [u'MOBI']
DEBUG: 0.1 marvin_edit_collections_cb: True
DEBUG: 0.1 marvin_protect_rb: False
DEBUG: 0.1 marvin_replace_rb: True
DEBUG: 0.1 marvin_update_rb: False
DEBUG: 0.1 preferred_reader_app: u'Marvin'
DEBUG: 0.1 iOSReaderApp:_get_connected_device_info()
DEBUG: 3.2 mounting 'com.appstafarian.MarvinIP'
DEBUG: 4.2 iOSReaderApp:_class_reconfigure('Marvin')
DEBUG: 4.2 iOSReaderApp:_load_reader_app_overlays('Marvin')
DEBUG: 4.3 loading 43 overlays
DEBUG: 4.3 adding overlay '_cover_subpath()'
DEBUG: 4.3 adding overlay '_cover_to_thumb()'
DEBUG: 4.3 adding overlay '_create_cover_element()'
DEBUG: 4.3 adding overlay '_create_new_book()'
DEBUG: 4.3 adding overlay '_evaluate_original_cover()'
DEBUG: 4.3 adding overlay '_evaluate_replaceable_cover()'
DEBUG: 4.3 adding overlay '_get_field_items()'
DEBUG: 4.3 adding overlay '_get_opf_tree()'
DEBUG: 4.3 adding overlay '_get_opf_xml()'
DEBUG: 4.3 adding overlay '_initialize_overlay()'
DEBUG: 4.3 adding overlay '_localize_database_path()'
DEBUG: 4.3 adding overlay '_parse_version()'
DEBUG: 4.3 adding overlay '_parse_xml()'
DEBUG: 4.3 adding overlay '_remove_existing_copy()'
DEBUG: 4.3 adding overlay '_report_upload_results()'
DEBUG: 4.3 adding overlay '_reset_ios_connection()'
DEBUG: 4.3 adding overlay '_schedule_metadata_update()'
DEBUG: 4.3 adding overlay '_stage_command_file()'
DEBUG: 4.3 adding overlay '_update_epub_metadata()'
DEBUG: 4.3 adding overlay '_wait_for_command_completion()'
DEBUG: 4.3 adding overlay '_watchdog_timed_out()'
DEBUG: 4.3 adding overlay '_xform_metadata_via_plugboard()'
DEBUG: 4.3 adding overlay 'add_books_to_metadata()'
DEBUG: 4.3 adding overlay 'books()'
DEBUG: 4.3 adding overlay 'can_handle()'
DEBUG: 4.3 adding overlay 'can_handle_windows()'
DEBUG: 4.3 adding overlay 'delete_books()'
DEBUG: 4.3 adding overlay 'eject()'
DEBUG: 4.3 adding overlay 'get_busy_flag()'
DEBUG: 4.3 adding overlay 'get_cc_mapping()'
DEBUG: 4.3 adding overlay 'get_file()'
DEBUG: 4.3 adding overlay 'guess_type()'
DEBUG: 4.3 adding overlay 'is_usb_connected()'
DEBUG: 4.3 adding overlay 'is_usb_connected_windows()'
DEBUG: 4.3 adding overlay 'post_yank_cleanup()'
DEBUG: 4.3 adding overlay 'prepare_addable_books()'
DEBUG: 4.3 adding overlay 'remove_books_from_metadata()'
DEBUG: 4.3 adding overlay 'set_busy_flag()'
DEBUG: 4.3 adding overlay 'set_cc_mapping()'
DEBUG: 4.3 adding overlay 'sync_booklists()'
DEBUG: 4.3 adding overlay 'thumbnail()'
DEBUG: 4.3 adding overlay 'upload_books()'
DEBUG: 4.3 adding overlay 'xml_to_unicode()'
DEBUG: 4.3 iOSReaderApp:_initialize_overlay(Marvin)
DEBUG: 4.3 existing thumb cache at '/Users/vladimyrh/Library/Caches/calibre/Marvin/thumbs.zip'
DEBUG: 4.3 ~~~~~ switching to Marvin overlay ~~~~~~
Starting up...
DEBUG: 6.8 MarvinManagerAction:genesis(v1.1.1)
DEBUG: 6.8 MarvinManagerAction:init_options()
DEBUG: 6.8 AnnotationsDB:connect(db_version: 1)
DEBUG: 6.8 MarvinManagerAction:inflate_dialog_resources()
DEBUG: 6.8 MarvinManagerAction:compile_ui()
DEBUG: 7.4 iOSReaderApp:startup()
DEBUG: 8.2 MarvinManagerAction:rebuild_menus()
DEBUG: 8.2 Marvin not connected
Started up in 3.80 seconds with 1400 books
DEBUG: 9.4 libiMobileDevice:_afc_get_file_inf* Notification delivered.
o( ERROR: Object not found (8) path:u'/Library/calibre/connected.xml')
DEBUG: 12.4 libiMobileDevice:_afc_get_file_info( ERROR: Object not found (8) path:u'/Library/calibre/connected.xml')
DEBUG: 15.5 libiMobileDevice:_afc_get_file_info( ERROR: Object not found (8) path:u'/Library/calibre/connected.xml')
DEBUG: 18.5 libiMobileDevice:_afc_get_file_info( ERROR: Object not found (8) path:u'/Library/calibre/connected.xml')
DEBUG: 21.5 libiMobileDevice:_afc_get_file_info( ERROR: Object not found (8) path:u'/Library/calibre/connected.xml')
DEBUG: 24.5 Marvin version: (2, 6, 0)
DEBUG: 24.5 iOSReaderApp:reset()
DEBUG: 24.5 detected_device = (1452L, 4779L, 832L, u'Apple Inc.', u'iPad', u'99c98607a112bf5deb0ff18d9de154f6b471234d')
DEBUG: 24.5 report_progress = <bound method DeviceManager.report_progress of <DeviceManager(Thread-5, started daemon 4661972992)>>
DEBUG: 24.5 iOSReaderApp:open()
DEBUG: 24.5 vid:05ac pid:12ab
DEBUG: 24.5 MarvinManagerAction:on_device_connection_changed(iOS reader applications)
DEBUG: 24.5 mounting com.appstafarian.MarvinIP
DEBUG: 25.8 MarvinManagerAction:on_device_connection_changed(iPad3,4 running iOS 7.0.4)
DEBUG: 25.8 has_password: False
DEBUG: 25.8 MarvinManagerAction:rebuild_menus()
DEBUG: 25.8 Marvin connected
DEBUG: 25.8 iOSReaderApp:set_progress_reporter()
Job: 1 Get device information started
DEBUG: 25.8 iOSReaderApp:get_device_information()
DEBUG: 25.8 DeviceClass : iPad
DEBUG: 25.8 DeviceColor : black
DEBUG: 25.8 DeviceName : Vladimyr’s iPad
DEBUG: 25.8 FSBlockSize : 4096
DEBUG: 25.8 FSFreeBytes : 4256112640
DEBUG: 25.8 FSTotalBytes : 29909303296
DEBUG: 25.8 FirmwareVersion : iBoot-1940.3.5
DEBUG: 25.8 HardwareModel : P101AP
DEBUG: 25.8 ModelNumber : MD511
DEBUG: 25.8 PasswordProtected : False
DEBUG: 25.8 ProductType : iPad3,4
DEBUG: 25.8 ProductVersion : 7.0.4
DEBUG: 25.8 SerialNumber : DMQJL9XPF183
DEBUG: 25.8 TimeIntervalSince1970: 1389670529.05
DEBUG: 25.8 TimeZone : America/Indiana/Indianapolis
DEBUG: 25.8 TimeZoneOffsetFromUTC: -18000.0
DEBUG: 25.8 UniqueDeviceID : 99c98607a112bf5deb0ff18d9de154f6b471234d
DEBUG: 25.8 iOSReaderApp:free_space()
DeviceJob: 1 Get device information done, calling callback
DeviceJob: 1 Get device information callback returned
DEBUG: 26.4 iOSReaderApp:set_progress_reporter()
Job: 2 Get list of books on device started
DEBUG: 26.4 iOSReaderApp:books()
DEBUG: 26.4 iOSReaderApp:_localize_database_path()
DEBUG: 26.4 updating local_db from u'/Library/mainDb.sqlite'
DEBUG: 26.5 (a series of cover files not found messages…)
DeviceJob: 2 Get list of books on device done, calling callback
DEBUG: 33.8 iOSReaderApp:settings(format_map for 'Marvin': [u'epub'])
DEBUG: 33.8 iOSReaderApp:settings(format_map for 'Marvin': [u'epub'])
DeviceJob: 2 Get list of books on device callback returned
Job: 2 Get list of books on device finished
No details available.
DEBUG: 54.0 iOSReaderApp:settings(format_map for 'Marvin': [u'epub'])
DEBUG: 55.9 iOSReaderApp:set_progress_reporter()
Job: 3 Upload 1 books to device:Harry Crow started
DEBUG: 55.9 iOSReaderApp:setplugboards()
Device using plugboard epub MARVIN None
Setting metadata in: Harry Crow at: /var/folders/7/fd1t8fms7cb0wjvmjy0dt1c80000gn/T/calibre_1.19.0_tmp_DeC6ZY/OtDui8caltmpfmt.epub
DEBUG: 56.9 iOSReaderApp:upload_books()
DEBUG: 56.9 'Harry Crow' by [u'robst'] does not exist in Marvin
DEBUG: 56.9 iOSReaderApp:_update_epub_metadata(Harry Crow)
DEBUG: 56.9 iOSReaderApp:_xform_metadata_via_plugboard(Harry Crow)
Device using plugboard epub MARVIN None
DEBUG: 56.9 no matching plugboard
DEBUG: 57.8 iOSReaderApp:_cover_to_thumb(Harry Crow)
DEBUG: 57.9 iOSReaderApp:_create_new_book(title: u'Harry Crow' uuid: u'5639ce11-ac64-4d7a-a7da-faa303ad951a')
DEBUG: 58.0 iOSReaderApp:_get_field_items(Harry Crow)
DEBUG: 58.0 iOSReaderApp:_evaluate_replaceable_cover()
DEBUG: 58.0 cover is replaceable
DEBUG: 58.1 iOSReaderApp:_stage_command_file(upload_books: 297 bytes)
DEBUG: 58.1 iOSReaderApp:_wait_for_command_completion(upload_books)
DEBUG: 58.1 22:36:39.012479: waiting for '/Library/calibre/status.xml'
DEBUG: 58.1 libiMobileDevice:_afc_get_file_info( ERROR: Object not found (8) path:u'/Library/calibre/status.xml')
DEBUG: 59.1 22:36:40.017363: monitoring progress of upload_books
DEBUG: 59.1 22:36:40.025684: 0 100%
DEBUG: 60.1 22:36:41.032048: 'upload_books' complete
DEBUG: 60.1 iOSReaderApp:_localize_database_path()
DEBUG: 60.1 MarvinManagerAction:marvin_status_changed(upload_books)
DEBUG: 60.1 updating local_db from u'/Library/mainDb.sqlite'
DeviceJob: 3Job: 3 Upload 1 books to device:Harry Crow done, calling callback
Upload 1 books to device:Harry Crow finished
No details available.
DEBUG: 60.2 iOSReaderApp:add_books_to_metadata()
DEBUG: 60.5 iOSReaderApp:set_plugboards()
DeviceJob: 3 Upload 1 books to device:Harry Crow callback returned
DEBUG: 60.5 iOSReaderApp:set_progress_reporter()
Job: 4 Send metadata to device started
DEBUG: 60.5 iOSReaderApp:sync_booklists(automatic metadata management disabled)
DEBUG: 60.5 iOSReaderApp:free_space()
DeviceJob: 4 Send metadata to device done, calling callback
DeviceJob: 4 Send metadata to device callback returned
Job: 3 Upload 1 books to device:Harry Crow finished
No details available.
Job: 4 Send metadata to device finished
No details available.
DEBUG: 64.0 MarvinManagerAction:rebuild_menus()
DEBUG: 64.0 Marvin connected
DEBUG: 65.8 MarvinManagerAction:show_installed_books()
DEBUG: 65.8 MarvinManagerAction:launch_library_scanner(updating library index for virtual library u'')
DEBUG: 65.8 MarvinManagerAction:start_library_indexing()
DEBUG: 65.8 MarvinManagerAction:_busy_panel_setup()
DEBUG: 66.6 MarvinManagerAction:library_index_complete()
DEBUG: 66.6 MarvinManagerAction:_busy_panel_teardown()
DEBUG: 66.7 BookStatusDialog:initialize()
DEBUG: 66.7 BookStatusDialog:_generate_booklist()
DEBUG: 67.3 BookStatusDialog:_scan_library_books(571)
DEBUG: 68.1 BookStatusDialog:_get_installed_books()
DEBUG: 68.1 BookStatusDialog:_scan_marvin_books(533 books)
DEBUG: 68.1 BookStatusDialog:_localize_hash_cache()
DEBUG: 68.1 remote hash cache: v1, 533 books in cache
DEBUG: 68.1 BookStatusDialog:_purge_cached_orphans()
DEBUG: 69.5 BookStatusDialog:_update_remote_hash_cache()
DEBUG: 70.8 BookStatusDialog:_get_cover_hash(a series of no cover available messages…)
DEBUG: 73.0 BookStatusDialog:_purge_cover_hash_orphans()
DEBUG: 73.0 BookStatusDialog:_busy_panel_setup(Removing obsolete cover hashes)
DEBUG: 73.7 BookStatusDialog:_busy_panel_teardown()
DEBUG: 73.7 BookStatusDialog:_generate_marvin_hash_map()
DEBUG: 73.7 BookStatusDialog:_find_fuzzy_matches()
DEBUG: 73.7 BookStatusDialog:_busy_panel_setup(Preparing Marvin library view…)
DEBUG: 74.4 BookStatusDialog:_construct_table_data()
DEBUG: 75.0 BookStatusDialog:_construct_table_view()
DEBUG: 76.5 BookStatusDialog:_update_refresh_button()
DEBUG: 76.5 BookStatusDialog:_busy_panel_teardown()
DEBUG: 76.5 MarvinManagerAction:show_installed_books(533 books)
DEBUG: 114.4 BookStatusDialog:dispatch_button_click()
DEBUG: 114.4 BookStatusDialog:accept()
DEBUG: 114.4 BookStatusDialog:_save_column_widths()
DEBUG: 120.3 MarvinManagerAction:shutting_down()
GRiker
commented
10 years ago @uten, please send a copy of this book (Harry Crow) to me.
Is it generally true that the books you're having problems with are fan fiction?
uten
commented
10 years ago Book sent. Also, while the majority is fanfic, that is only because most of my books are fanfic.. But there are also novels, and non-fiction with the same problem.
And there is no common link. Some of the fanfic were created by FLAG, calibre's fanfic downloader, various other websites, and even some I've converted. Same with the Novels and Non-Fiction, they all come from various different places, including smashwords.
GRiker
commented
10 years ago @uten, thanks for sending the book. I added it to my test library, then added it to Marvin using calibre. Upon opening MXD it properly displayed as green when Show match status is enabled. I examined the book to be sure that there was nothing unusual about its internal metadata.
In studying your debug log more closely, it appears that you have a total of 1400 books in your calibre library (reported just after DEBUG 8.2). At DEBUG 67.3, it reports that you have 571 EPUBS in your library, and at DEBUG 68.1, it reports that you have 533 books installed in Marvin.
One possible explanation is that you have some filtering applied to your calibre library so that only certain books are displayed at the time you launch MXD. MXD only evaluates books currently displayed in your calibre window when showing matches. This was an optimization added for users with very large libraries.
Can you confirm that before launching MXD, Harry Crow is visible in the main calibre window?
Are you using a virtual library, or is there any filter that would prevent the Harry Crow book from being listed in the main calibre window at the time MXD is launched?
uten
commented
10 years ago I do use virtual Libraries. Specifically, the one I was using at the time is entitled "Books for Marvin" and contains all epub books. I did verify that the book was showing. It also shows a tick in calibre. The remaining epubs that I have still to transfer from calibre need to have metadata entered, or something similar.
That was the first thing I checked.
uten
commented
10 years ago Also, while the book shows as white, view metadata shows the Marvin metadata and the calibre metadata. The only difference is it doesn't have the option underneath each window to transfer metadata.
But selecting multiple white files does allow me to export metadata from calibre, turning the books green. Closing the explorer, then reloading it shows the books as being white again.
The few yellow books allow me to export metadata from calibre, turning the books green. Closing the explorer, then reloading it shows the books as still green. HOWEVER, Disconnecting via the calibre connector in Marvin, then reconnecting, causes the books to turn yellow again, and the metadata shows up as the old bad stuff again.
So, since the metadata and status change is obviously not being saved on my ipad, is it possible that the data saying the book is in calibre isn't being written either? Is that stored on the ipad?
GRiker
commented
10 years ago Please do an experiment to update the metadata on the iPad from calibre for a given book, then opening the book in Marvin to see if the metadata was actually updated from the calibre values, regardless of the colors shown by the plugin.
Instead of reporting "the metadata shows up as the old bad stuff again", it would be more helpful if you could report the actual before/after metadata.
Also, can you do a manual inspection of the books in Marvin to see if there are duplicates of the problem books?
GRiker
commented
10 years ago @uten, please try this as well.
In calibre, open Preferences|Advanced|Miscellaneous, then click Open calibre configuration directory.
Exit calibre.
In that directory, open the plugins folder, then locate the file Marvin XD.json.
Open that file with a text editor, and find the line "show_match_colors": true,
Copy the entire line, and paste a copy below it, making sure that the indentation is exactly the same for the pasted copy.
Edit that line to read "development_mode": true,
Save and close the file.
This will add some additional diagnostic information to the debug log.
In that same directory, open iOS reader applications.json
Find the line "debug_plugin": true,
Change true to false
Save and close the file.
That will turn off debugging for the iOSRA plugin.
Do the same sequence you did before, deleting the book in Marvin, then starting Calibre in debug mode, adding the book, starting MXD. After the book shows white, close calibre, post the debug log.
uten
commented
10 years ago I chose Harry Crow to alter the metadata. I removed all the tags. Which turned it yellow. Updated the metadata which turned it white. The metadata did update however and it has stayed updated. Also picked a green colored book. Worked correctly. Green > Yellow > Green..
Weird thing number one. With yellow books. Just updating the metadata causes the following. They turn green until you disconnect, then next time you connect, they are yellow. However, one of them with this problem had an author written "Brighton, Patrick". I switched it to "Patrick Brighton" in calibre, then updated the metadata to Marvin. The book is now green. Since that worked, I altered the author of Harry Crow and updated. Still white.
One thing I did notice. Just looking at a books metadata in calibre MIGHT turn the book yellow. I chose the "Edit Metadata Individually" and then selecting next (or cancel). Some books turned yellow from that, and some didn't. After choosing to update the metadata to Marvin, they would switch back to either white or green. But it wasn't every book, and it wasn't just white or green colored books. That happening made me go WTF?? :)
uten
commented
10 years ago Debug File is way toooooooo long now, so I'll email it to you.
uten
commented
10 years ago BTW, what checksum format do you use to determine the hash? I figured I could run some files through to see if they match, but the only thing I know so far is what type it isn't.. :)
GRiker
commented
10 years ago The checksum is computed by generating an MD5 hash of all the text and CSS files, ignoring images and the OPF file. The objective is to uniquely identify the content of the book, so that matches may be identified when the metadata doesn't match. The source code is here, starting at line #2837.
GRiker
commented
10 years ago I will review the debug file, but in the meantime, please try this:
Any differences?
GRiker
commented
10 years ago (Edited version of debug log follows)
calibre 1.19 isfrozen: True is64bit: True
Darwin-13.0.0-x86_64-i386-64bit Darwin ('64bit', '')
('Darwin', '13.0.0', 'Darwin Kernel Version 13.0.0: Thu Sep 19 22:22:27 PDT 2013; root:xnu-2422.1.72~6/RELEASE_X86_64')
Python 2.7.5
OSX: ('10.9.1', ('', '', ''), 'x86_64')
Starting up...
DEBUG: 0.0 MarvinManagerAction:genesis(v1.1.1)
DEBUG: 0.0 MarvinManagerAction:init_options()
DEBUG: 0.0 AnnotationsDB:connect(db_version: 1)
DEBUG: 0.0 MarvinManagerAction:inflate_dialog_resources()
DEBUG: 0.0 MarvinManagerAction:compile_ui()
DEBUG: 1.2 MarvinManagerAction:rebuild_menus()
DEBUG: 1.2 Marvin not connected
Started up in 3.62 seconds with 1418 books
DEBUG: 3.1 MarvinManagerAction:on_device_connection_changed(iOS reader applications)
DEBUG: 3.1 mounting com.appstafarian.MarvinIP
DEBUG: 4.4 MarvinManagerAction:on_device_connection_changed(iPad3,4 running iOS 7.0.4)
DEBUG: 4.4 has_password: False
DEBUG: 4.4 MarvinManagerAction:rebuild_menus()
DEBUG: 4.4 Marvin connected
Job: 1 Get device information started
DeviceJob: 1 Get device information done, calling callback
DeviceJob: 1 Get device information callback returned
Job: 2 Get list of books on device started
Job: 1 Get device information finished
No details available.
DeviceJob: 2 Get list of books on device done, calling callback
DeviceJob: 2 Get list of books on device callback returned
Job: 2 Get list of books on device finished
No details available.
Job: 3 Upload 1 books to device:Harry Crow started
Device using plugboard epub MARVIN None
Setting metadata in: Harry Crow at: /var/folders/7_/fd1t8fms7cb0wjvmjy0dt1c80000gn/T/calibre_1.19.0_tmp_CtPqdv/aijWd3caltmpfmt.epub
Device using plugboard epub MARVIN None
DEBUG: 49.9 MarvinManagerAction:marvin_status_changed(upload_books)
DeviceJob: 3 Upload 1 books to device:Harry Crow done, calling callback
DeviceJob: 3 Upload 1 books to device:Harry Crow callback returned
Job: 4 Send metadata to device started
DeviceJob: 4 Send metadata to device done, calling callback
DeviceJob: 4 Send metadata to device callback returned
Job: 3 Upload 1 books to device:Harry Crow finished
No details available.
Job: 4 Send metadata to device finished
No details available.
DEBUG: 61.4 MarvinManagerAction:main_menu_button_clicked()
DEBUG: 61.4 MarvinManagerAction:show_installed_books()
DEBUG: 61.4 MarvinManagerAction:launch_library_scanner(updating library index for virtual library u'Books For Marvin')
DEBUG: 61.4 MarvinManagerAction:start_library_indexing()
DEBUG: 61.4 MarvinManagerAction:_busy_panel_setup()
DEBUG: 62.2 MarvinManagerAction:library_index_complete()
DEBUG: 62.2 MarvinManagerAction:_busy_panel_teardown()
DEBUG: 62.3 BookStatusDialog:initialize()
DEBUG: 62.3 BookStatusDialog:_generate_booklist()
DEBUG: 62.9 BookStatusDialog:_scan_library_books(589)
DEBUG: 63.7 BookStatusDialog:_get_installed_books()
DEBUG: 63.7 BookStatusDialog:_scan_marvin_books(533 books)
DEBUG: 63.7 BookStatusDialog:_localize_hash_cache()
DEBUG: 63.7 remote hash cache: v1, 533 books in cache
DEBUG: 65.1 BookStatusDialog:_update_remote_hash_cache()
DEBUG: 65.8 BookStatusDialog:_get_calibre_id(u"1,001 Old-Time Household Hints: Timeless Bits of Household Wisdom for Today's Home and Garden" u'The Editors Of Yankee Magazine')
DEBUG: 65.8 UUID match
DEBUG: 65.8 BookStatusDialog:_get_on_device_status()
(multiple calls to _get_calibre_id, _get_on_device_status)
DEBUG: 70.7 BookStatusDialog:_get_calibre_id(u'Harry Crow' u'Robst')
DEBUG: 70.7 UUID match
DEBUG: 70.7 BookStatusDialog:_get_on_device_status()
DEBUG: 70.7 533 cached books from Marvin
(multiple books)
DEBUG: 70.7 Harry Crow pubdate: datetime.datetime(2012, 6, 5, 4, 0)
DEBUG: 70.7 BookStatusDialog:_generate_marvin_hash_map()
DEBUG: 70.7 BookStatusDialog:_find_fuzzy_matches()
DEBUG: 70.8 BookStatusDialog:_busy_panel_setup(Preparing Marvin library view…)
DEBUG: 71.4 BookStatusDialog:_construct_table_data()
DEBUG: 71.4 BookStatusDialog:_generate_match_quality('1,001 Old-Time Household Hints: Timeless Bits of Household Wisdom for Today's Home and Garden')
DEBUG: 71.4 uuid: u'4cf5de09-e5c5-4e68-8bd7-f7184b110e8e'
DEBUG: 71.4 matches: [u'4cf5de09-e5c5-4e68-8bd7-f7184b110e8e']
DEBUG: 71.4 on_device: u'Main'
DEBUG: 71.4 hash: '31b96dc9fdc9635f1598027818b8389a'
DEBUG: 71.4 metadata_mismatches: {}
DEBUG: 71.4 match_quality: GREEN
(multiple calls to _generate_match_quality)
DEBUG: 72.2 BookStatusDialog:_generate_match_quality('Harry Crow')
DEBUG: 72.2 uuid: u'5639ce11-ac64-4d7a-a7da-faa303ad951a'
DEBUG: 72.2 matches: []
DEBUG: 72.2 on_device: u'Main'
DEBUG: 72.2 hash: '0b77c34ae22b35f84dabc50b0c702c7c'
DEBUG: 72.2 metadata_mismatches: {}
DEBUG: 72.2 match_quality: WHITE
DEBUG: 72.2 BookStatusDialog:_construct_table_view()
DEBUG: 74.1 BookStatusDialog:_update_refresh_button()
DEBUG: 74.1 BookStatusDialog:_busy_panel_teardown()
DEBUG: 74.1 MarvinManagerAction:show_installed_books(533 books)
DEBUG: 95.0 BookStatusDialog:dispatch_button_click()
DEBUG: 95.0 BookStatusDialog:accept()
DEBUG: 95.0 BookStatusDialog:_save_column_widths()
DEBUG: 101.5 MarvinManagerAction:shutting_down() Deleting the hashes has taken my white books from around 250 to 13. Main problem fixed. Something was obviously wrong with the hashes.
While fixing up some of the last thirteen white books, I did discover something. It seems that editing the book in calibre, then updating it to calibre left the book white. I tried just deleting the calibre hash info. Still White. On deleting both hash sets, a refresh turned the book green. Also happened that way if I just deleted the Marvin hash info. How does the plugin handle it if the hash it has is different to the files? Does it check for that? Is it possible the plugin has old hash data?
I do have a couple of ebooks that have a different problem. The cover shows up in Calibre, and in Marvin, but not on Marvin XD. Even after deleting the book, and re-adding it (after polishing of course), the problem remains. And causes the book to be white. I'll send you a copy of the ebook to see if it shows up for you.
Thanks so much for all your help so far!
GRiker
commented
10 years ago @uten, I've built an updated version of MXD with more diagnostics that should help pin down the remaining unmatched books. You can download it from this link.
Save it to your desktop.
Start calibre with your iPad disconnected, then install the downloaded plugin from Preferences|Advanced|Plugins.
Create a debug log again with the same settings for development_mode (true).
This version of MXD should show the unmatched books in gray. Please confirm that the unmatched books are shown in gray, and that the other books are displayed with the proper colors.
The debug log will be large, email it to me as you have before.
If you're having different issues with MXD, please open a new issue with those details. I will try loading the book you sent me to see if there's an obvious problem with it.
In answer to your question about stored hashes, deleting the hashes from calibre and Marvin clears them completely, then they are recreated then next time you run the plugin. Calibre library hashes are stored in calibre's db, Marvin's hashes are stored on the iDevice.
In general, is the common denominator with the books that are missing that you've polished them in calibre and sent them to Marvin again after polishing?
GRiker
commented
10 years ago FYI, the book you sent me The Trouble With Families loaded into Marvin and displayed in MXD without issue, so it's probably related to this issue. No need to open a new one.
A couple of questions:
uten
commented
10 years ago I sorta figured the problem was related - hence why I mentioned it in this thread. In hindsight, using the word different wasn't the best idea as by different, I meant the cover issue only.
BTW, when you viewed the metadata of the book I sent you, did it show the cover on the copy that was sent to Marvin??
As for Polish, I only use the Update Metadata, and Update Cover options.
I use the Replace option When Sending Books already in Marvin, as I discovered early on that it wasn't sending updated epubs to Marvin.
I'll send you the log as soon as I can. Thanks again!
GRiker
commented
10 years ago With respect to the book you sent, MXD is not showing the cover in the metadata dialog, although the cover does display properly in Marvin's library view, calibre's library view and calibre's device view. The book is shown as green in MXD. There's clearly something wrong with the cover not showing in MXD. I'll track that down.
GRiker
commented
10 years ago The problem with the missing covers in the MXD metadata comparison dialog is now fixed. It was actually a typo in the iOS reader applications plugin, so you will need to download a revised version of that from this link.
That fix will not affect any of the matching color issues we've been working on in this issue. Thanks for spotting that.
uten
commented
10 years ago Ok, Debug file on the way.
Sorry it took so long, but I had a migraine. :(
The book used is the one I sent you - "The trouble with families" by LaraBr
GRiker
commented
10 years ago (Edited version of debug log focusing on missing books only)
calibre 1.19 isfrozen: True is64bit: True
Darwin-13.0.0-x86_64-i386-64bit Darwin ('64bit', '')
('Darwin', '13.0.0', 'Darwin Kernel Version 13.0.0: Thu Sep 19 22:22:27 PDT 2013; root:xnu-2422.1.72~6/RELEASE_X86_64')
Python 2.7.5
OSX: ('10.9.1', ('', '', ''), 'x86_64')
Starting up...
DEBUG: 0.0 MarvinManagerAction:genesis(v1.1.1)
DEBUG: 0.0 MarvinManagerAction:init_options()
DEBUG: 0.0 AnnotationsDB:connect(db_version: 1)
DEBUG: 0.0 MarvinManagerAction:inflate_dialog_resources()
DEBUG: 0.0 MarvinManagerAction:compile_ui()
DEBUG: 1.2 MarvinManagerAction:rebuild_menus()
DEBUG: 1.2 Marvin not connected
Started up in 3.55 seconds with 1418 books
DEBUG: 17.7 MarvinManagerAction:on_device_connection_changed(iOS reader applications)
DEBUG: 17.7 mounting com.appstafarian.MarvinIP
DEBUG: 19.0 MarvinManagerAction:on_device_connection_changed(iPad3,4 running iOS 7.0.4)
DEBUG: 19.0 has_password: False
DEBUG: 19.0 MarvinManagerAction:rebuild_menus()
DEBUG: 19.0 Marvin connected
Job: 1 Get device information started
DeviceJob: 1 Get device information done, calling callback
DeviceJob: 1 Get device information callback returned
Job: 2 Get list of books on device started
DEBUG: 19.7 libiMobileDevice:_afc_get_file_info( ERROR: Object not found (8) path:u'/Library/Application Support/com.appstafarian.marvin.covers/060d72fb9d676f0d9c657e4a7bf59d37.jpg')
Job: 1 Get device information finished
No details available.
DeviceJob: 2 Get list of books on device done, calling callback
DeviceJob: 2 Get list of books on device callback returned
Job: 2 Get list of books on device finished
No details available.
Job: 3 Upload 1 books to device:The trouble with families started
Device using plugboard epub MARVIN None
Setting metadata in: The trouble with families at: /var/folders/7_/fd1t8fms7cb0wjvmjy0dt1c80000gn/T/calibre_1.19.0_tmp_hbxlNJ/iN0fzVcaltmpfmt.epub
Device using plugboard epub MARVIN None
DEBUG: 43.9 MarvinManagerAction:marvin_status_changed(upload_books)
DeviceJob: 3 Upload 1 books to device:The trouble with families done, calling callback
DeviceJob: 3 Upload 1 books to device:The trouble with families callback returned
Job: 4 Send metadata to device started
DeviceJob: 4 Send metadata to device done, calling callback
DeviceJob: 4 Send metadata to device callback returned
Job: 4 Send metadata to device finished
No details available.
Job: 3 Upload 1 books to device:The trouble with families finished
No details available.
DEBUG: 48.7 MarvinManagerAction:main_menu_button_clicked()
DEBUG: 48.7 MarvinManagerAction:show_installed_books()
DEBUG: 48.7 MarvinManagerAction:launch_library_scanner(updating library index for virtual library u'Books For Marvin')
DEBUG: 48.7 MarvinManagerAction:start_library_indexing()
DEBUG: 48.7 MarvinManagerAction:_busy_panel_setup()
DEBUG: 49.4 MarvinManagerAction:library_index_complete()
DEBUG: 49.4 MarvinManagerAction:_busy_panel_teardown()
DEBUG: 49.5 BookStatusDialog:initialize()
DEBUG: 49.5 BookStatusDialog:_generate_booklist()
DEBUG: 50.2 BookStatusDialog:_scan_library_books(589)
(Edited to include only GRAY books)
DEBUG: 50.9 HASH UUID
DEBUG: 50.9 First Heart c9a62e91fc32a7ec1b0bedf01e6eb1ca [u'176f7a1e-0ae0-4dc7-b777-2b7f58d69f30']
DEBUG: 50.9 Gifts of the Goddess f2346f4bdc4a4b44dbdb8eaab294e5e3 [u'29c4d913-2a47-4a80-8a69-575a9c21d8ae', u'579d66d8-a2fa-420c-abb6-330388ec9ed3']
DEBUG: 50.9 Gothic Fascination 95d62d898ef0f307b7432d2601e096b7 [u'b93e0ed9-7184-49b3-af9f-efa0cf67d6b2', u'528562b3-2bdd-4444-8743-4254c6ba81e4']
DEBUG: 50.9 H Santini 2a91f50df53830c6075c59bf7d198c06 [u'41c7db38-ab38-4af3-8fc7-58262585ea26', u'd944ec9b-554d-478f-9fe6-560d027dd1d8']
DEBUG: 50.9 Harry Potter and the Go… 913ab342dc4484e6f7ca2265c27e7cfb [u'71dfe648-aec0-46eb-8c33-67433892ba51', u'1977fa15-0031-4de2-b17e-c125754977ba']
DEBUG: 50.9 Healing db5e0af5e1dca8c83964dd0a96d22043 [u'6bc7c6f6-ce9d-446a-91a5-eae488de9d57', u'661367c7-50c2-43b2-bc96-875547515e42']
DEBUG: 50.9 Hidden Pasts 6c0e1895725601c07c5c69a9d34f74f4 [u'b1ad37bf-8b76-4b3c-ba10-d62211198594', u'0002bd11-2b5c-4489-af92-6ab6b4ca3ca8']
DEBUG: 50.9 Knight By Inheritance 03c7f1dd109ddea01b7bda2df6970851 [u'5c3b761c-c884-4810-bf1e-758ae079c7c3', u'2044bb78-0d5c-48e0-8707-cdc39a25d664']
DEBUG: 50.9 Outrageous Fortune e5644b2e23c5573b7ed4c4824fa670d2 [u'0981655d-7f25-4961-9b57-2711d3aae1b7', u'520e4f87-4ac0-4bae-b01c-b6f2422647c2']
DEBUG: 50.9 Rose's are Red, Laven… 12dc7972ac8c2faad61a449a3a5251a0 [u'f35be59d-09c0-4c13-a128-0c1a727282cc', u'8cb2b6c3-0045-4ad0-8817-19c31580b25a']
DEBUG: 50.9 The Trouble With Fam… a3d859e84cbdca2fae1d18c2bd2b5d45 [u'dd39a3c6-ddb3-49fe-a84a-3f6e64f9d9f9']
DEBUG: 50.9 Waiting for Sunrise d5bcdc5f49acb7f3444d569ede3415bd [u'f8a14dd9-c606-43d4-92a6-c8609dacb22e']
DEBUG: 50.9 BookStatusDialog:_get_installed_books()
DEBUG: 50.9 BookStatusDialog:_scan_marvin_books(533 books)
DEBUG: 50.9 BookStatusDialog:_localize_hash_cache()
DEBUG: 51.0 remote hash cache: v1, 533 books in cache
DEBUG: 52.2 BookStatusDialog:_update_remote_hash_cache()
DEBUG: 54.2 BookStatusDialog:_get_calibre_id(u'First Heart' u'groggy')
DEBUG: 54.2 UUID match: 176f7a1e-0ae0-4dc7-b777-2b7f58d69f30
DEBUG: 54.2 BookStatusDialog:_get_calibre_id(u'Gifts of the Goddess' u'Lucifael')
DEBUG: 54.2 UUID match: 579d66d8-a2fa-420c-abb6-330388ec9ed3
DEBUG: 54.3 BookStatusDialog:_get_calibre_id(u'Gothic Fascination' u'Lucifael')
DEBUG: 54.3 UUID match: 528562b3-2bdd-4444-8743-4254c6ba81e4
DEBUG: 54.3 BookStatusDialog:_get_calibre_id(u'H Santini, Lock Stock and Two Smoking Wands' u'Lucifael')
DEBUG: 54.3 UUID match: 41c7db38-ab38-4af3-8fc7-58262585ea26
DEBUG: 54.3 BookStatusDialog:_get_calibre_id(u'Harry Potter and the Golden Snitch' u'Lucifael')
DEBUG: 54.3 UUID match: 71dfe648-aec0-46eb-8c33-67433892ba51
DEBUG: 54.4 BookStatusDialog:_get_calibre_id(u'Healing' u'Lucifael')
DEBUG: 54.4 UUID match: 661367c7-50c2-43b2-bc96-875547515e42
DEBUG: 54.5 BookStatusDialog:_get_calibre_id(u'Hidden Pasts' u'Lucifael')
DEBUG: 54.5 UUID match: 0002bd11-2b5c-4489-af92-6ab6b4ca3ca8
DEBUG: 54.6 BookStatusDialog:_get_calibre_id(u'Knight By Inheritance' u'Lucifael')
DEBUG: 54.6 UUID match: 5c3b761c-c884-4810-bf1e-758ae079c7c3
DEBUG: 54.9 BookStatusDialog:_get_calibre_id(u'Outrageous Fortune' u'Lucifael')
DEBUG: 54.9 UUID match: 0981655d-7f25-4961-9b57-2711d3aae1b7
DEBUG: 55.1 BookStatusDialog:_get_calibre_id(u"Rose's are Red, Lavender is Blue" u'Lucifael')
DEBUG: 55.1 UUID match: 8cb2b6c3-0045-4ad0-8817-19c31580b25a
DEBUG: 55.5 BookStatusDialog:_get_calibre_id(u'Waiting for Sunrise' u'OlBear')
DEBUG: 55.5 UUID match: f8a14dd9-c606-43d4-92a6-c8609dacb22e
DEBUG: 55.7 BookStatusDialog:_get_calibre_id(u'The trouble with families' u'LaraBr')
DEBUG: 55.7 UUID match: dd39a3c6-ddb3-49fe-a84a-3f6e64f9d9f9
DEBUG: 55.8 BookStatusDialog:_generate_marvin_hash_map()
DEBUG: 55.8 BookStatusDialog:_find_fuzzy_matches()
DEBUG: 55.8 BookStatusDialog:_busy_panel_setup(Preparing Marvin library view…)
DEBUG: 56.4 BookStatusDialog:_construct_table_data()
DEBUG: 56.7 BookStatusDialog:_generate_match_quality('First Heart')
DEBUG: 56.7 uuid: u'176f7a1e-0ae0-4dc7-b777-2b7f58d69f30'
DEBUG: 56.7 matches: []
DEBUG: 56.7 on_device: u'Main'
DEBUG: 56.7 hash: '45c934923a07661d1c5b9ae00eb7e3bd'
DEBUG: 56.7 metadata_mismatches: {}
DEBUG: 56.7 match_quality: GRAY
DEBUG: 56.7 BookStatusDialog:_generate_match_quality('Gifts of the Goddess')
DEBUG: 56.7 uuid: u'579d66d8-a2fa-420c-abb6-330388ec9ed3'
DEBUG: 56.7 matches: [u'29c4d913-2a47-4a80-8a69-575a9c21d8ae', u'579d66d8-a2fa-420c-abb6-330388ec9ed3']
DEBUG: 56.7 on_device: u'Main'
DEBUG: 56.7 hash: 'f2346f4bdc4a4b44dbdb8eaab294e5e3'
DEBUG: 56.7 metadata_mismatches: {}
DEBUG: 56.7 match_quality: GRAY
DEBUG: 56.7 BookStatusDialog:_generate_match_quality('Gothic Fascination')
DEBUG: 56.7 uuid: u'528562b3-2bdd-4444-8743-4254c6ba81e4'
DEBUG: 56.7 matches: [u'b93e0ed9-7184-49b3-af9f-efa0cf67d6b2', u'528562b3-2bdd-4444-8743-4254c6ba81e4']
DEBUG: 56.7 on_device: u'Main'
DEBUG: 56.7 hash: '95d62d898ef0f307b7432d2601e096b7'
DEBUG: 56.7 metadata_mismatches: {}
DEBUG: 56.7 match_quality: GRAY
DEBUG: 56.7 BookStatusDialog:_generate_match_quality('H Santini, Lock Stock and Two Smoking Wands')
DEBUG: 56.7 uuid: u'41c7db38-ab38-4af3-8fc7-58262585ea26'
DEBUG: 56.7 matches: [u'41c7db38-ab38-4af3-8fc7-58262585ea26', u'd944ec9b-554d-478f-9fe6-560d027dd1d8']
DEBUG: 56.7 on_device: u'Main'
DEBUG: 56.7 hash: '2a91f50df53830c6075c59bf7d198c06'
DEBUG: 56.7 metadata_mismatches: {}
DEBUG: 56.7 match_quality: GRAY
DEBUG: 56.7 BookStatusDialog:_generate_match_quality('Harry Potter and the Golden Snitch')
DEBUG: 56.7 uuid: u'71dfe648-aec0-46eb-8c33-67433892ba51'
DEBUG: 56.7 matches: [u'71dfe648-aec0-46eb-8c33-67433892ba51', u'1977fa15-0031-4de2-b17e-c125754977ba']
DEBUG: 56.7 on_device: u'Main'
DEBUG: 56.7 hash: '913ab342dc4484e6f7ca2265c27e7cfb'
DEBUG: 56.7 metadata_mismatches: {}
DEBUG: 56.7 match_quality: GRAY
DEBUG: 56.7 BookStatusDialog:_generate_match_quality('Healing')
DEBUG: 56.7 uuid: u'661367c7-50c2-43b2-bc96-875547515e42'
DEBUG: 56.7 matches: [u'6bc7c6f6-ce9d-446a-91a5-eae488de9d57', u'661367c7-50c2-43b2-bc96-875547515e42']
DEBUG: 56.7 on_device: u'Main'
DEBUG: 56.7 hash: 'db5e0af5e1dca8c83964dd0a96d22043'
DEBUG: 56.7 metadata_mismatches: {}
DEBUG: 56.7 match_quality: GRAY
DEBUG: 56.7 BookStatusDialog:_generate_match_quality('Hidden Pasts')
DEBUG: 56.7 uuid: u'0002bd11-2b5c-4489-af92-6ab6b4ca3ca8'
DEBUG: 56.7 matches: [u'b1ad37bf-8b76-4b3c-ba10-d62211198594', u'0002bd11-2b5c-4489-af92-6ab6b4ca3ca8']
DEBUG: 56.7 on_device: u'Main'
DEBUG: 56.7 hash: '6c0e1895725601c07c5c69a9d34f74f4'
DEBUG: 56.7 metadata_mismatches: {}
DEBUG: 56.7 match_quality: GRAY
DEBUG: 56.8 BookStatusDialog:_generate_match_quality('Knight By Inheritance')
DEBUG: 56.8 uuid: u'5c3b761c-c884-4810-bf1e-758ae079c7c3'
DEBUG: 56.8 matches: [u'5c3b761c-c884-4810-bf1e-758ae079c7c3', u'2044bb78-0d5c-48e0-8707-cdc39a25d664']
DEBUG: 56.8 on_device: u'Main'
DEBUG: 56.8 hash: '03c7f1dd109ddea01b7bda2df6970851'
DEBUG: 56.8 metadata_mismatches: {}
DEBUG: 56.8 match_quality: GRAY
DEBUG: 56.9 BookStatusDialog:_generate_match_quality('Outrageous Fortune')
DEBUG: 56.9 uuid: u'0981655d-7f25-4961-9b57-2711d3aae1b7'
DEBUG: 56.9 matches: [u'0981655d-7f25-4961-9b57-2711d3aae1b7', u'520e4f87-4ac0-4bae-b01c-b6f2422647c2']
DEBUG: 56.9 on_device: u'Main'
DEBUG: 56.9 hash: 'e5644b2e23c5573b7ed4c4824fa670d2'
DEBUG: 56.9 metadata_mismatches: {}
DEBUG: 56.9 match_quality: GRAY
DEBUG: 56.9 BookStatusDialog:_generate_match_quality('Rose's are Red, Lavender is Blue')
DEBUG: 56.9 uuid: u'8cb2b6c3-0045-4ad0-8817-19c31580b25a'
DEBUG: 56.9 matches: [u'f35be59d-09c0-4c13-a128-0c1a727282cc', u'8cb2b6c3-0045-4ad0-8817-19c31580b25a']
DEBUG: 56.9 on_device: u'Main'
DEBUG: 56.9 hash: '12dc7972ac8c2faad61a449a3a5251a0'
DEBUG: 56.9 metadata_mismatches: {}
DEBUG: 56.9 match_quality: GRAY
DEBUG: 57.1 BookStatusDialog:_generate_match_quality('Waiting for Sunrise')
DEBUG: 57.1 uuid: u'f8a14dd9-c606-43d4-92a6-c8609dacb22e'
DEBUG: 57.1 matches: []
DEBUG: 57.1 on_device: u'Main'
DEBUG: 57.1 hash: '1925248fd59de81ce71b3d3ae7c4f375'
DEBUG: 57.1 metadata_mismatches: {}
DEBUG: 57.1 match_quality: GRAY
DEBUG: 57.1 BookStatusDialog:_generate_match_quality('The trouble with families')
DEBUG: 57.1 uuid: u'dd39a3c6-ddb3-49fe-a84a-3f6e64f9d9f9'
DEBUG: 57.1 matches: []
DEBUG: 57.1 on_device: u'Main'
DEBUG: 57.1 hash: '0cb70a005d7bf3ca2f9b5e4701d98a93'
DEBUG: 57.1 metadata_mismatches: {}
DEBUG: 57.1 match_quality: GRAY
DEBUG: 57.2 BookStatusDialog:_construct_table_view()
DEBUG: 58.6 BookStatusDialog:_update_refresh_button()
DEBUG: 58.6 BookStatusDialog:_busy_panel_teardown()
DEBUG: 58.6 MarvinManagerAction:show_installed_books(533 books)
DEBUG: 69.5 BookStatusDialog:dispatch_button_click()
DEBUG: 69.5 BookStatusDialog:accept()
DEBUG: 69.5 BookStatusDialog:_save_column_widths()
DEBUG: 74.0 MarvinManagerAction:shutting_down() @uten, here's what I'm observing from the latest debug log.
There are 12 unmatched books. 9 of those 12 - the ones with multiple UUID entries in the debug log - should have been flagged with a different color. I've added pink to represent books meeting the following conditions:
I've updated MXD. Please download and install it from this link, then confirm that there are 9 pink books and 3 gray books displayed.
The other 3 - First Heart, The Trouble With Families, Waiting for Sunrise don't match because the hashes don't match between calibre and Marvin. We will need to do some more experiments to isolate why the hashes aren't matching.
Please confirm that the updated plugin behaves as described, then we'll consider the next step.
uten
commented
10 years ago Ok, I'll leave my calibre library as is, so it can be tested. Downloaded and installed it MXD. Error when attempting to view the Marvin Library.
calibre, version 1.19.0 ERROR: Unhandled exception: IndexError:list index out of range
calibre 1.19 isfrozen: True is64bit: True Darwin-13.0.0-x86_64-i386-64bit Darwin ('64bit', '') ('Darwin', '13.0.0', 'Darwin Kernel Version 13.0.0: Thu Sep 19 22:22:27 PDT 2013; root:xnu-2422.1.72~6/RELEASE_X86_64') Python 2.7.5 OSX: ('10.9.1', ('', '', ''), 'x86_64') Traceback (most recent call last): File "calibre_plugins.marvin_manager.action", line 423, in main_menu_button_clicked File "calibre_plugins.marvin_manager.action", line 773, in show_installed_books File "calibre_plugins.marvin_manager.book_status", line 1300, in initialize File "calibre_plugins.marvin_manager.book_status", line 3160, in _construct_table_data File "calibre_plugins.marvin_manager.book_status", line 3079, in _generate_match_quality IndexError: list index out of range
GRiker
commented
10 years ago I've posted another build of MXD that reports duplicate content in your calibre library.
If that works properly, we can focus on the three GRAY books.
GRiker
commented
10 years ago @uten, with respect to the three titles displayed in GRAY, are there any common denominators between these books? Have you polished or edited them recently?
uten
commented
10 years ago Installed. Worked no problem.
10 Books are Pink (the 9 plus the yellow book we hadn't been focusing on). 3 are Gray.
The interesting thing about the pink ones, is that it has picked up books the duplicate finder didn't. Finding and removing the duplicates solves the problem. The three gray books were downloaded using the fanfic downloader, along with four others, as a group, and are the most recent books added to calibre. Out of the seven, two had covers added, and along with two more, were then polished, and added to Marvin. Three of those four are gray. The fourth is fine (one of the two with covers). If it was the polish or download, I'd expect the fourth one to be gray as well.
One of those, The Trouble with Families, I sent to you and you had no problems with.
The yellow book is a little strange. It is in the pink, but has no uuid duplicate, verified by looking in the debug file.
However, both author name and author sort were written "LastName, FirstName" (eg. "Smith, John") Correcting that to (AN: John Smith - AS: Smith, John) and resending the book fixed it.. I did try just resending the book. Didn't work.
GRiker
commented
10 years ago Well, we're making progress.
I made a slight tweak to the color algorithm to select yellow before pink. You can download that update to MXD from this link.
Let's focus on the gray books now, and we might as well use The Trouble With Families again. Please do this:
What color is that book?
uten
commented
10 years ago We have a winner!!!!
Not only did "The trouble with Families" turn green, so did the other two gray books! Awesome!
GRiker
commented
10 years ago That tells us that by clearing the hashes and allowing the plugin to rescan both the calibre library and Marvin library, the books were recognized as being a match. However, it would still be useful to understand what process causes the books to become unmatched. If you could continue to pay attention to what changes you're making to your books, and what happens when you download them, that would be helpful. If you can create a repeatable test case where a book fails to match, please let me know.
Thanks for your persistence.
uten
commented
10 years ago Will do.
One other issue with the colors.. I just cleared all the pink colored books. When I load MarvinXD, it is still finding a single duplicate. But all books are green. When I deleted that one, it found another. Then it found two. All duplicates are now deleted but for some reason, it missed the last few..
GRiker
commented
10 years ago The check for calibre duplicates was ignoring whether or not the book was installed in Marvin. Since the plugin is focused on Marvin users, I revised the check for duplicates to confirm that the book is both duplicated in calibre and installed in Marvin. You can download the updated MXD from this link.
uten
commented
10 years ago Updated the plugin.. I believe I have discovered the problem that causes the gray books, but I need to check a few final things to verify that.
Unfortunately, there is a problem.. When I right click on any book in the MarvinXD window, regardless if it is Green, Gray or Yellow, this happens.
calibre, version 1.19.0 ERROR: Unhandled exception: AttributeError:'BookStatusDialog' object has no attribute 'YELLOW'
calibre 1.19 isfrozen: True is64bit: True Darwin-13.0.0-x86_64-i386-64bit Darwin ('64bit', '') ('Darwin', '13.0.0', 'Darwin Kernel Version 13.0.0: Thu Sep 19 22:22:27 PDT 2013; root:xnu-2422.1.72~6/RELEASE_X86_64') Python 2.7.5 OSX: ('10.9.1', ('', '', ''), 'x86_64') Traceback (most recent call last): File "calibre_plugins.marvin_manager.book_status", line 363, in contextMenuEvent AttributeError: 'BookStatusDialog' object has no attribute 'YELLOW'
GRiker
commented
10 years ago I've posted an updated version of MXD fixing that error. You can download it from this link.
uten
commented
10 years ago I think I've got it.
I discovered that updating any fanfic, caused it to become gray, even if the updated version was sent to Marvin.
To verify that it wasn't the fanfic downloader, I added a single word to the title page of a book (using sigil), then updated the book to Marvin. It becomes gray.
I also did the above step using Calibre's built in ebook editor, with the same result.
I also tested it by adding the same word to another book, and then restarted calibre, instead of sending it. That book also became gray. Sending the book to Marvin didn't fix that.
Rebuilding the hashes will fix MOST of the errors. I had three gray books that stayed gray even after deletion and replacing in Marvin. To turn it green, I had to remove the book in the Marvin UI, restart calibre, rebuild the hashes, resend the book to Marvin.
However, in all cases, you can view the calibre books metadata, showing it is still linked correctly in that part of the plugin.
So, the most likely cause is either the UUID, the hash, or some other identifier you use, is not updating when a book is edited (Do you check for edited/updated books??), or is only partially updated. Also possible, some identifier/data is being kept after a book is deleted, as the above glitch possibly shows.
I might be off base with the cause, but hopefully you can now get the error to repeat.
Hope that helps!!!
GRiker
commented
10 years ago Is The Trouble With Families a fanfic book? Is it a book that exhibits the problem for you?
uten
commented
10 years ago It is, and it does when updated.
However, the same occurs with editing ANY book. I added a word to chapter one of a novel, and it turned gray.
EDIT - Actually, I should have said, ANY book that has already been sent to Marvin.
GRiker
commented
10 years ago I think I have fixed the problem. Being able to reproduce it was very useful, thanks.
You will need to download and install updated versions of both iOS reader applications and Marvin XD.
Please test this as throughly as you've been testing the other releases. I needed to make some significant changes to the signaling between the plugins, hopefully I didn't break anything along the way.
GRiker
commented
10 years ago One additional note - I revised the hash algorithm to make it even more paranoid, which means that some of your previously matched books may show as gray. If that happens, delete both the calibre hashes and the Marvin hashes. You should only need to do this once.
I tested by editing a book with calibre's editor, then sending it to Marvin, then opening MXD.
GRiker
commented
10 years ago I just spotted one more problem, I'll post here when I have another release available.
GRiker
commented
10 years ago Here's what I observed after minimal testing with the most recently posted plugins, please confirm:
If you edit the book with calibre's editor, then send the edited book to Marvin, then open MXD, the book is displayed as green.
If you edit the book with Sigil, then send the edited book to Marvin, then open MXD, the book is displayed as gray.
The reason for this behavior is that calibre maintains a last_modified timestamp for the book which is updated a book's metadata is edited, or the content is edited with calibre's editor. However, using Sigil to edit the book does not cause the last_modified field to be updated. If the plugin doesn't know about the updated content, the hash won't be recalculated.
Therefore, I think this means that if your workflow is edit->download-MXD, the edit phase must be done with calibre's editor. Using an external editor like Sigil (even though it's dispatched from calibre) would require either deleting the hashes or deleting the book in Marvin then reloading.
Let me know what you find.
uten
commented
10 years ago Whew, looks like the problem is licked, except for one spot..
All of these are with the book already loaded in Marvin.
If you edit the book with calibre's editor, then send the edited book to Marvin. then open MXD, the book is displayed as Green.
If you edit the book with calibre's editor, then "forget" to send the edited book to Marvin. then open MXD, the book is displayed as Gray. Closing MXD, re-sending the book to Marvin, and then opening MXD turns the book Green.
If you update a fanfic, then send the updated book to Marvin, then open MXD, the book is displayed as Green.
If you update a fanfic, then "forget" to send it to Marvin, then open MXD, the book is displayed as Yellow. Choosing 'update Metadata to Marvin' turns the book Green. This is an ERROR. Closing MXD, and then re-opening it, the book is now Gray. Closing MXD, resending the book to Marvin, and then opening MXD turns the book Green.
If you update a cover in calibre, or edit metadata in calibre, then open MXD, the book is displayed as Yellow. Choosing update Metadata to Marvin turns the book Green.
If you edit the book with Sigil, then send the edited book to Marvin, then open MXD, the book is displayed as Gray. If you then close MXD, then resend it to Marvin, then open Marvin, the book is displayed as Green. Not once did I need to delete the hash libraries and the step where you had to delete the book in Marvin is now no longer needed. :)
Tested each with multiple books to make sure each wasn't a fluke. :)
Hope it helps. And thanks again for working so hard to find the errors.
GRiker
commented
10 years ago If you update a fanfic, then "forget" to send it to Marvin, then open MXD, the book is displayed as Yellow. Choosing 'update Metadata to Marvin' turns the book Green. This is an ERROR. Closing MXD, and then re-opening it, the book is now Gray. Closing MXD, resending the book to Marvin, and then opening MXD turns the book Green.
I tried this with both The Trouble With Families and Harry Crow and didn't see this behavior. Here's what happened for me:
If you have a particular book where this happens consistently, please send it to me.
Also, please create a debug log where this happens and send it to me.
The attached screenshot shows several eBooks as missing from calibre.
The Cassandra Clare books are in a different calibre library so that is correct. The others however, do exist in the current calibre library.
Or have I misunderstood something?
ps Very best wishes for a great 2014. :)