Get List of books on Device Fails

[Dhardrian]

Marvin version: 2.7 iDevice: iPad Model ME392LL/A Calibre version: 1.33.0 Marvin XD plugin version: 1.2.1 iOS reading applications plugin version: 1.3.8 Operating system: Windows 7, 64-bit iTunes version: 11.1.5.5

The specific steps to recreate the problem: Attach iPad: Run Marvin: Select Get Books: Select Calibre

Description of the problem: On connection, "Get Device Information" finishes, and then "Get List of books on Device begins". When it reaches 100% it display's an error: "Error Communicating with Device"

iPad contains a Marvin library with 10930 books.

Error Dialog:

calibre, version 1.33.0 ERROR: Error: Error communicating with device

Traceback (most recent call last): File "site-packages\calibre\gui2\device.py", line 86, in run File "site-packages\calibre\gui2\device.py", line 490, in _books File "C:\Users\Rob\AppData\Local\Temp\calibre_liq3mx\ezk2vb_ios_local_db\Marvin_overlays.py", line 381, in books File "C:\Users\Rob\AppData\Local\Temp\calibre_liq3mx\ezk2vb_ios_local_db\Marvin_overlays.py", line 2314, in _snapshot_booklist File "jsoninit.py", line 250, in dumps File "json\encoder.py", line 210, in encode MemoryError

Debug Listing:

calibre 1.33 isfrozen: True is64bit: False Windows-7-6.1.7601-SP1 Windows ('32bit', 'WindowsPE') 32bit process running on 64bit windows ('Windows', '7', '6.1.7601') Python 2.7.5 Windows: ('7', '6.1.7601', 'SP1', 'Multiprocessor Free')

[GRiker]

I'm looking at it. The problem is triggered with larger libraries. I test with a library of 4200 books, which is not enough to trigger it.

I'll let you know when I have something for you to test.

[GRiker]

You can bypass the error for now by disabling caching in the iOSRA plugin:

• Exit calibre
• disconnect your iPad
• open a command window
• Paste the following line into the command window:

calibre-debug -c "from calibre.utils.config import JSONConfig; JSONConfig('plugins/iOS reader applications').set('booklist_caching',False)"

• Press return to execute the command
• Reconnect your iPad
• Restart calibre

[Dhardrian]

That did get past the error I had been receiving, however an attempt to update (Send to device, with settings in iOSRA set for Marvin to update metadata, not overwrite) on the entire library in the device failed with the following error dialog:

[cid:image001.png@01CF5BD7.D7B98670]

I was not running in debug mode at the time, so I do not have a debug log. I am attempting to repeat with debug enabled.

Rob

From: GRiker [mailto:notifications@github.com] Sent: Saturday, April 19, 2014 10:50 AM To: GRiker/calibre-marvin-manager Cc: Robert Mohr Subject: Re: [calibre-marvin-manager] Get List of books on Device Fails (#46)

You can bypass the error for now by disabling caching in the iOSRA plugin:

• Exit calibre
• disconnect your iPad
• open a command window
• Paste the following line into the command window:

calibre-debug -c "from calibre.utils.config import JSONConfig; JSONConfig('plugins/iOS reader applications').set('booklist_caching',False)"

• Press return to execute the command
• Reconnect your iPad
• Restart calibre

— Reply to this email directly or view it on GitHubhttps://github.com/GRiker/calibre-marvin-manager/issues/46#issuecomment-40875852.

[GRiker]

First, we need to resolve the caching issue with your large library. I've created a beta build which you can download from this link.

• Exit calibre
• Disconnect your iPad
• Download that zip file to your desktop
• Follow the directions to install a plugin manually from this link.

Next, let's re-enable caching. Execute the code below in a command window:

calibre-debug -c "from calibre.utils.config import JSONConfig; JSONConfig('plugins/iOS reader applications').set('booklist_caching',True)"

• Connect your iPad
• Start Marvin, then start the calibre connector
• Start calibre in debug mode
• The first time you run it, it will take longer as it rebuilds the cache.
• After initialization has completed, click the Jobs spinner in the lower right corner, noting the Running time for the Get list of books on device line. It took 1:36 for my library of 4200 books, I would expect yours to be proportionally larger.
• Exit calibre, then restart.
• Click on the Jobs spinner and wait for the Get list of books on device job to complete.
• Please note the Running time for that job. It should be quicker - for my 4200 book library, loading time with the cache was 0:31, a 3:1 improvement.
• Please post those times in a new comment.
• If that all works as expected, then please close this ticket.
• If there's a new problem, please post the debug log in a new comment.

With respect to you sending 10,000 books at once, that's a separate issue.

If you need to open a new ticket addressing a different problem with respect to sending huge blocks of books, please open a ticket at the iOSRA issue tracker.

Let me know how it goes.

[Dhardrian]

After 3 minutes, during the caching phase, the progress bar went from 99%, briefly to 1%, then immediately to 100% and then the error dialog “error communicating with device” occurred:

---

calibre, version 1.33.0 ERROR: Error: Error communicating with device

Traceback (most recent call last): File "site-packages\calibre\gui2\device.py", line 86, in run File "site-packages\calibre\gui2\device.py", line 490, in _books File "C:\Users\Rob\AppData\Local\Temp\calibre_t6wi85\gg0tiw_ios_local_db\Marvin_overlays.py", line 393, in books File "C:\Users\Rob\AppData\Local\Temp\calibre_t6wi85\gg0tiw_ios_local_db\Marvin_overlays.py", line 2470, in _snapshot_booklist File "site-packages\calibre\devices\idevice\libimobiledevice.py", line 238, in copy_to_idevice MemoryError

Debug log attached.

From: GRiker [mailto:notifications@github.com] Sent: Sunday, April 20, 2014 7:59 AM To: GRiker/calibre-marvin-manager Cc: Robert Mohr Subject: Re: [calibre-marvin-manager] Get List of books on Device Fails (#46)

First, we need to resolve the caching issue with your large library. I've created a beta build which you can download from this linkhttps://www.dropbox.com/s/vaihbwks17sfdz4/iOS%20reader%20applications.zip.

• Exit calibre
• Disconnect your iPad
• Download that zip file to your desktop
• Follow the directions to install a plugin manually from this linkhttp://www.mobileread.com/forums/showthread.php?t=118680.

Next, let's re-enable caching. Execute the code below in a command window:

calibre-debug -c "from calibre.utils.config import JSONConfig; JSONConfig('plugins/iOS reader applications').set('booklist_caching',True)"

• Connect your iPad
• Start Marvin, then start the calibre connector
• Start calibre in debug mode
• The first time you run it, it will take longer as it rebuilds the cache.
• After initialization has completed, click the Jobs spinner in the lower right corner, noting the Running time for the Get list of books on device line. It took 1:36 for my library of 4200 books, I would expect yours to be proportionally larger.
• Exit calibre, then restart.
• Click on the Jobs spinner and wait for the Get list of books on device job to complete.
• Please note the Running time for that job. It should be quicker - for my 4200 book library, loading time with the cache was 0:31, a 3:1 improvement.
• Please post those times in a new comment.
• If that all works as expected, then please close this ticket.
• If there's a new problem, please post the debug log in a new comment.

With respect to you sending 10,000 books at once, that's a separate issue.

If you need to open a new ticket addressing a different problem with respect to sending huge blocks of books, please open a ticket at the iOSRA issue trackerhttps://github.com/GRiker/calibre-ios-reader-applications/issues?state=open.

Let me know how it goes.

— Reply to this email directly or view it on GitHubhttps://github.com/GRiker/calibre-marvin-manager/issues/46#issuecomment-40896601.

calibre 1.33 isfrozen: True is64bit: False Windows-7-6.1.7601-SP1 Windows ('32bit', 'WindowsPE') 32bit process running on 64bit windows ('Windows', '7', '6.1.7601') Python 2.7.5 Windows: ('7', '6.1.7601', 'SP1', 'Multiprocessor Free') Starting up... DEBUG: 0.0 MarvinManagerAction:genesis(v1.2.1) DEBUG: 0.0 MarvinManagerAction:init_options() DEBUG: 0.0 AnnotationsDB:connect(db_version: 1) DEBUG: 0.0 MarvinManagerAction:inflate_dialog_resources() DEBUG: 0.1 MarvinManagerAction:compile_ui() DEBUG: 18.5 MarvinManagerAction:rebuild_menus() DEBUG: 18.5 Marvin not connected Started up in 43.28 seconds with 10930 books DEBUG: 30.1 MarvinManagerAction:on_device_connection_changed(iOS reader applications) DEBUG: 30.1 mounting com.appstafarian.MarvinIP DEBUG: 31.3 iPad3,4 running iOS 7.1 DEBUG: 31.3 has_password: True DEBUG: 31.3 MarvinManagerAction:rebuild_menus() DEBUG: 31.3 Marvin connected Job: 1 Get device information started DeviceJob: 1 Get device information done, calling callback Job: 1 Get device information finished No details available. DeviceJob: 1 Get device information callback returned Job: 2 Get list of books on device started Job: 1 Get device information finished No details available. DEBUG: 33.1 libiMobileDevice:_afc_get_file_info( ERROR: Object not found (8) path:u'/Library/calibre.mm/booklist.db') afc_receive_data: entire_len is larger than MAXIMUM_PACKET_SIZE, (384065 > 65536)!DeviceJob: 2 Get list of books on device done, calling callback

Traceback (most recent call last): File "site-packages\calibre\gui2\device.py", line 86, in run File "site-packages\calibre\gui2\device.py", line 490, in _books File "C:\Users\Rob\AppData\Local\Temp\calibre_t6wi85\gg0tiw_ios_local_db\Marvin_overlays.py", line 393, in books File "C:\Users\Rob\AppData\Local\Temp\calibre_t6wi85\gg0tiw_ios_local_db\Marvin_overlays.py", line 2470, in _snapshot_booklist File "site-packages\calibre\devices\idevice\libimobiledevice.py", line 238, in copy_to_idevice MemoryError

DeviceJob: 2 Get list of books on device callback returned Job: 2 Get list of books on device finished

Traceback (most recent call last):
  File "site-packages\calibre\gui2\device.py", line 86, in run
  File "site-packages\calibre\gui2\device.py", line 490, in _books
  File "C:\Users\Rob\AppData\Local\Temp\calibre_t6wi85\gg0tiw_ios_local_db\Marvin_overlays.py", line 393, in books
  File "C:\Users\Rob\AppData\Local\Temp\calibre_t6wi85\gg0tiw_ios_local_db\Marvin_overlays.py", line 2470, in _snapshot_booklist
  File "site-packages\calibre\devices\idevice\libimobiledevice.py", line 238, in copy_to_idevice
MemoryError

DEBUG: 255.6 MarvinManagerAction:shutting_down()

[Dhardrian]

I am headed to Europe in the AM. Back Saturday the 25th.

From: GRiker [mailto:notifications@github.com] Sent: Sunday, April 20, 2014 7:59 AM To: GRiker/calibre-marvin-manager Cc: Robert Mohr Subject: Re: [calibre-marvin-manager] Get List of books on Device Fails (#46)

First, we need to resolve the caching issue with your large library. I've created a beta build which you can download from this linkhttps://www.dropbox.com/s/vaihbwks17sfdz4/iOS%20reader%20applications.zip.

• Exit calibre
• Disconnect your iPad
• Download that zip file to your desktop
• Follow the directions to install a plugin manually from this linkhttp://www.mobileread.com/forums/showthread.php?t=118680.

Next, let's re-enable caching. Execute the code below in a command window:

calibre-debug -c "from calibre.utils.config import JSONConfig; JSONConfig('plugins/iOS reader applications').set('booklist_caching',True)"

• Connect your iPad
• Start Marvin, then start the calibre connector
• Start calibre in debug mode
• The first time you run it, it will take longer as it rebuilds the cache.
• After initialization has completed, click the Jobs spinner in the lower right corner, noting the Running time for the Get list of books on device line. It took 1:36 for my library of 4200 books, I would expect yours to be proportionally larger.
• Exit calibre, then restart.
• Click on the Jobs spinner and wait for the Get list of books on device job to complete.
• Please note the Running time for that job. It should be quicker - for my 4200 book library, loading time with the cache was 0:31, a 3:1 improvement.
• Please post those times in a new comment.
• If that all works as expected, then please close this ticket.
• If there's a new problem, please post the debug log in a new comment.

With respect to you sending 10,000 books at once, that's a separate issue.

If you need to open a new ticket addressing a different problem with respect to sending huge blocks of books, please open a ticket at the iOSRA issue trackerhttps://github.com/GRiker/calibre-ios-reader-applications/issues?state=open.

Let me know how it goes.

— Reply to this email directly or view it on GitHubhttps://github.com/GRiker/calibre-marvin-manager/issues/46#issuecomment-40896601.

[GRiker]

I think I have this resolved. When convenient, please download and install an updated beta from this link.

I believe that the problem is that the size of the cache file the plugin is trying to store is larger than the amount of available space on your iPad. I've added some code to limit the size of the cache that can be stored in the iOSRA plugin config dialog. It defaults to 1% of available space, with a max of 10%.

In your case, that should prevent the error, as it will automatically disable cache creation. Please let me know if this works as expected when you have a chance.

[GRiker]

I made another optimization to the caching which should help quite a bit. With my 4200 book test library, the initial load time is 1:35. The best-case load time for the next startup is now :08, a 12x improvement. I will be interested to hear your timings.

[GRiker]

There is a beta version of MXD available from this link.

[GRiker]

I've resolved some problems with caching that may or may not have affected you. I'd like you to update your versions of iOSRA and MXD to newer versions.

• Download the latest iOSRA plugin from this link
• Download the latest MXD plugin from this link
• Disconnect your iPad
• Update both plugins
• After updating both plugins, exit calibre
• Reconnect your iPad
• Start Marvin and the calibre connector
• Start calibre.
• Wait for initialization to complete
• Press and hold the Alt/Option key, then click the down arrow to the right of the blue MXD icon
• Click Developer
• From the Developer submenu, click Delete iOSRA booklist cache
• Exit and restart calibre

The initial load will take longer as the cache is rebuilt.

Please advise of any issues or problems.

[Dhardrian]

I am back from Europe, I can try this tonight.

Sent from my iPad

On Apr 28, 2014, at 5:22 AM, "GRiker" notifications@github.com<mailto:notifications@github.com> wrote:

I've resolved some problems with caching that may or may not have affected you. I'd like you to update your versions of iOSRA and MXD to newer versions.

• Download the latest iOSRA plugin from this linkhttps://www.dropbox.com/s/vaihbwks17sfdz4/iOS%20reader%20applications.zip
• Download the latest MXD plugin from this linkhttps://www.dropbox.com/s/six0c50asmvkjm8/Marvin%20XD.zip
• Disconnect your iPad
• Update both plugins
• After updating both plugins, exit calibre
• Reconnect your iPad
• Start Marvin and the calibre connector
• Start calibre.
• Wait for initialization to complete
• Press and hold the Alt/Option key, then click the down arrow to the right of the blue MXD icon
• Click Developer
• From the Developer submenu, click Delete iOSRA booklist cache
• Exit and restart calibre

The initial load will take longer as the cache is rebuilt.

Please advise of any issues or problems.

— Reply to this email directly or view it on GitHubhttps://github.com/GRiker/calibre-marvin-manager/issues/46#issuecomment-41551918.

[WyndhamMisio]

No issues.

I will now resume the other testing (timings).

[Dhardrian]

The error remains the same with the most recent plugin’s downloaded from the links below. Error dialog:

calibre, version 1.34.0 ERROR: Error: Error communicating with device

Traceback (most recent call last): File "site-packages\calibre\gui2\device.py", line 86, in run File "site-packages\calibre\gui2\device.py", line 490, in _books File "C:\Users\Rob\AppData\Local\Temp\calibre_js5ojt\kihie1_ios_local_db\Marvin_overlays.py", line 400, in books File "C:\Users\Rob\AppData\Local\Temp\calibre_js5ojt\kihie1_ios_local_db\Marvin_overlays.py", line 2585, in _snapshot_booklist File "site-packages\calibre\devices\idevice\libimobiledevice.py", line 238, in copy_to_idevice MemoryError

I have not yet tried the beta code you sent on the 21st and the 25th while I was traveling. Are both of those changes incorporated in these plugin versions?

From: GRiker [mailto:notifications@github.com] Sent: Monday, April 28, 2014 5:25 AM To: GRiker/calibre-marvin-manager Cc: Robert Mohr Subject: Re: [calibre-marvin-manager] Get List of books on Device Fails (#46)

I've resolved some problems with caching that may or may not have affected you. I'd like you to update your versions of iOSRA and MXD to newer versions.

• Download the latest iOSRA plugin from this linkhttps://www.dropbox.com/s/vaihbwks17sfdz4/iOS%20reader%20applications.zip
• Download the latest MXD plugin from this linkhttps://www.dropbox.com/s/six0c50asmvkjm8/Marvin%20XD.zip
• Disconnect your iPad
• Update both plugins
• After updating both plugins, exit calibre
• Reconnect your iPad
• Start Marvin and the calibre connector
• Start calibre.
• Wait for initialization to complete
• Press and hold the Alt/Option key, then click the down arrow to the right of the blue MXD icon
• Click Developer
• From the Developer submenu, click Delete iOSRA booklist cache
• Exit and restart calibre

The initial load will take longer as the cache is rebuilt.

Please advise of any issues or problems.

— Reply to this email directly or view it on GitHubhttps://github.com/GRiker/calibre-marvin-manager/issues/46#issuecomment-41551918.

[GRiker]

Please update to the latest versions and try again.

[Dhardrian]

Greg,

I updated to the versions in your email from yesterday, iOSRA version 1.3.9 and Marvin XD 1.2.2. Were there other versions you wanted me to try?

Rob

From: GRiker [mailto:notifications@github.com] Sent: Tuesday, April 29, 2014 4:26 PM To: GRiker/calibre-marvin-manager Cc: Robert Mohr Subject: Re: [calibre-marvin-manager] Get List of books on Device Fails (#46)

Please update to the latest versions and try again.

— Reply to this email directly or view it on GitHubhttps://github.com/GRiker/calibre-marvin-manager/issues/46#issuecomment-41744595.

[GRiker]

To clarify, you have downloaded from the links I provided in the last day? If not, please download and install from the provided links.

If so, please enable debug logging for iOSRA and create a debug log. You can find the instructions for enabling debug logging for iOSRA here.

Also, please check to see how much available memory you have on your iPad. You can find that in Settings | Usage - at the top there are two numbers, 'Available' and 'Used'. What are those numbers?

[Dhardrian]

Greg,

Correct, I downloaded from the links provided yesterday. Debug log attached. The error is occurring at the beginning of the device read now, not at the end.

Memory statistics for my iPad are:

Available Used 73.2 GB 42.3 GB

Rob

From: GRiker [mailto:notifications@github.com] Sent: Tuesday, April 29, 2014 4:35 PM To: GRiker/calibre-marvin-manager Cc: Robert Mohr Subject: Re: [calibre-marvin-manager] Get List of books on Device Fails (#46)

To clarify, you have downloaded from the links I provided in the last day? If not, please download and install from the provided links.

If so, please enable debug logging for iOSRA and create a debug log. You can find the instructions for enabling debug logging for iOSRA herehttps://github.com/GRiker/calibre-ios-reader-applications/issues/1.

Also, please check to see how much available memory you have on your iPad. You can find that in Settings | Usage - at the top there are two numbers, 'Available' and 'Used'. What are those numbers?

— Reply to this email directly or view it on GitHubhttps://github.com/GRiker/calibre-marvin-manager/issues/46#issuecomment-41745192.

calibre 1.34 isfrozen: True is64bit: False Windows-7-6.1.7601-SP1 Windows ('32bit', 'WindowsPE') 32bit process running on 64bit windows ('Windows', '7', '6.1.7601') Python 2.7.5 Windows: ('7', '6.1.7601', 'SP1', 'Multiprocessor Free') DEBUG: 0.0 iOSReaderApp:initialize(v1.3.9) DEBUG: 0.0 iOSReaderApp:compile_ui() DEBUG: 0.1 iOSReaderApp:_init_prefs(prefs created under v1.3.8) DEBUG: 0.1 booklist_caching: True DEBUG: 0.1 cc_mappings: {u'My Library': {u'marvin_word_count': {u'field': u'#mm_word_count', u'combobox': u'Word count'}, u'marvin_collections': {u'field': u'#genre', u'combobox': u'Genre'}}} DEBUG: 0.1 debug_libimobiledevice: False DEBUG: 0.1 debug_plugin: True DEBUG: 0.1 development_mode: False DEBUG: 0.1 device_booklist_cache_limit: 10 DEBUG: 0.1 device_booklist_caching: True DEBUG: 0.1 kindle_enabled_formats: [u'MOBI'] DEBUG: 0.1 kindle_supported_formats: [u'MOBI'] DEBUG: 0.1 marvin_edit_collections_cb: True DEBUG: 0.1 marvin_protect_rb: False DEBUG: 0.1 marvin_replace_rb: False DEBUG: 0.1 marvin_update_rb: True DEBUG: 0.1 plugin_diagnostics: True DEBUG: 0.1 preferred_reader_app: u'Marvin' DEBUG: 0.1 iOSReaderApp:_get_connected_device_info() DEBUG: 4.5 mounting 'com.appstafarian.MarvinIP' DEBUG: 5.6 iOSReaderApp:_class_reconfigure('Marvin') DEBUG: 5.6 iOSReaderApp:_load_reader_app_overlays('Marvin') DEBUG: 5.7 loading 58 overlays DEBUG: 5.7 adding overlay '_compare_mainDb_profiles()' DEBUG: 5.7 adding overlay '_cover_subpath()' DEBUG: 5.7 adding overlay '_cover_to_thumb()' DEBUG: 5.7 adding overlay '_create_cover_element()' DEBUG: 5.7 adding overlay '_create_empty_booklist_db()' DEBUG: 5.7 adding overlay '_create_new_book()' DEBUG: 5.7 adding overlay '_dehydrate_booklist()' DEBUG: 5.7 adding overlay '_establish_local_booklist_db_path()' DEBUG: 5.7 adding overlay '_evaluate_original_cover()' DEBUG: 5.7 adding overlay '_evaluate_replaceable_cover()' DEBUG: 5.7 adding overlay '_get_field_items()' DEBUG: 5.7 adding overlay '_get_opf_tree()' DEBUG: 5.7 adding overlay '_get_opf_xml()' DEBUG: 5.7 adding overlay '_initialize_overlay()' DEBUG: 5.7 adding overlay '_localize_booklist_db()' DEBUG: 5.7 adding overlay '_localize_database_path()' DEBUG: 5.7 adding overlay '_parse_version()' DEBUG: 5.7 adding overlay '_parse_xml()' DEBUG: 5.7 adding overlay '_profile_db()' DEBUG: 5.7 adding overlay '_rehydrate_booklist()' DEBUG: 5.7 adding overlay '_remove_existing_copy()' DEBUG: 5.7 adding overlay '_report_upload_results()' DEBUG: 5.7 adding overlay '_reset_ios_connection()' DEBUG: 5.7 adding overlay '_restore_from_snapshot()' DEBUG: 5.7 adding overlay '_schedule_metadata_update()' DEBUG: 5.7 adding overlay '_snapshot_booklist()' DEBUG: 5.7 adding overlay '_stage_command_file()' DEBUG: 5.7 adding overlay '_update_epub_metadata()' DEBUG: 5.7 adding overlay '_validate_dehydrated_booklist()' DEBUG: 5.7 adding overlay '_wait_for_command_completion()' DEBUG: 5.7 adding overlay '_watchdog_timed_out()' DEBUG: 5.7 adding overlay '_xform_metadata_via_plugboard()' DEBUG: 5.7 adding overlay 'add_books_to_metadata()' DEBUG: 5.7 adding overlay 'books()' DEBUG: 5.7 adding overlay 'can_handle()' DEBUG: 5.7 adding overlay 'can_handle_windows()' DEBUG: 5.7 adding overlay 'delete_books()' DEBUG: 5.7 adding overlay 'eject()' DEBUG: 5.7 adding overlay 'from_json()' DEBUG: 5.7 adding overlay 'get_busy_flag()' DEBUG: 5.7 adding overlay 'get_cc_mapping()' DEBUG: 5.7 adding overlay 'get_file()' DEBUG: 5.7 adding overlay 'guess_type()' DEBUG: 5.7 adding overlay 'is_usb_connected()' DEBUG: 5.7 adding overlay 'is_usb_connected_windows()' DEBUG: 5.7 adding overlay 'post_yank_cleanup()' DEBUG: 5.7 adding overlay 'prepare_addable_books()' DEBUG: 5.7 adding overlay 'remove_books_from_metadata()' DEBUG: 5.7 adding overlay 'set_busy_flag()' DEBUG: 5.7 adding overlay 'set_cc_mapping()' DEBUG: 5.7 adding overlay 'shutdown()' DEBUG: 5.7 adding overlay 'sort_key()' DEBUG: 5.7 adding overlay 'startup()' DEBUG: 5.7 adding overlay 'sync_booklists()' DEBUG: 5.7 adding overlay 'thumbnail()' DEBUG: 5.7 adding overlay 'to_json()' DEBUG: 5.7 adding overlay 'upload_books()' DEBUG: 5.7 adding overlay 'xml_to_unicode()' DEBUG: 5.7 iOSReaderApp:_initialize_overlay(Marvin) DEBUG: 5.7 existing thumb cache at 'C:\Users\Rob\AppData\Local\calibre-cache\Marvin\thumbs.zip' DEBUG: 5.7 ~~~~~ switching to Marvin overlay ~~~~~~ Starting up... DEBUG: 16.2 MarvinManagerAction:genesis(v1.2.2) DEBUG: 16.2 MarvinManagerAction:init_options() DEBUG: 16.2 AnnotationsDB:connect(db_version: 1) DEBUG: 16.2 MarvinManagerAction:inflate_dialog_resources() DEBUG: 16.3 MarvinManagerAction:compile_ui() DEBUG: 16.6 iOSReaderApp:startup() DEBUG: 16.6 Waiting for calibre connector... DEBUG: 18.8 Marvin version: (2, 7, 0) DEBUG: 19.8 iOSReaderApp:reset() DEBUG: 19.8 iOSReaderApp:open() DEBUG: 19.8 Vendor ID (vid):05ac Product ID: (pid):12ab DEBUG: 28.2 MarvinManagerAction:rebuild_menus() DEBUG: 28.2 Marvin not connected Started up in 31.72 seconds with 10930 books DEBUG: 37.8 MarvinManagerAction:on_device_connection_changed(iOS reader applications) DEBUG: 37.8 mounting com.appstafarian.MarvinIP DEBUG: 38.9 iPad3,4 running iOS 7.1.1 DEBUG: 38.9 has_password: True DEBUG: 38.9 MarvinManagerAction:rebuild_menus() DEBUG: 38.9 Marvin connected DEBUG: 38.9 iOSReaderApp:set_progress_reporter() Job: 1 Get device information started DEBUG: 38.9 iOSReaderApp:get_device_information() DEBUG: 38.9 DeviceClass : iPad DEBUG: 38.9 DeviceColor : black DEBUG: 38.9 DeviceName : Rob Mohr’s iPad DEBUG: 38.9 FSBlockSize : 4096 DEBUG: 38.9 FSFreeBytes : 78,809,268,224 DEBUG: 38.9 FSTotalBytes : 123,994,841,088 DEBUG: 38.9 FirmwareVersion : iBoot-1940.10.58 DEBUG: 38.9 HardwareModel : P101AP DEBUG: 38.9 ModelNumber : ME392 DEBUG: 38.9 PasswordProtected : False DEBUG: 38.9 ProductType : iPad3,4 DEBUG: 38.9 ProductVersion : 7.1.1 DEBUG: 38.9 SerialNumber : DMPK5B3TFCYF DEBUG: 38.9 TimeIntervalSince1970: 1398815192.75 DEBUG: 38.9 TimeZone : America/Los_Angeles DEBUG: 38.9 TimeZoneOffsetFromUTC: -25200.0 DEBUG: 38.9 UniqueDeviceID : f86c1193a1c728e07d3e88f6f80694e8d9224bd4 DEBUG: 38.9 iOSReaderApp:free_space() DeviceJob: 1 Get device information done, calling callback Job: 1 Get device information finished No details available. DeviceJob: 1 Get device information callback returned DEBUG: 39.4 iOSReaderApp:set_progress_reporter() Job: 2 Get list of books on device started DEBUG: 39.4 iOSReaderApp:books() DEBUG: 39.4 iOSReaderApp:_localize_database_path() Job: 1 Get device information finished No details available. DEBUG: 39.4 copying local_db from u'/Library/mainDb.sqlite' DEBUG: 40.4 iOSReaderApp:_restore_from_snapshot() DEBUG: 40.4 Analyzing local booklist DEBUG: 40.9 pushing booklist.db to remote_cache_folder DeviceJob: 2 Get list of books on device done, calling callback DeviceJob: 2 Get list of books on device callback returned Job: 2 Get list of books on device finished

Traceback (most recent call last):
  File "site-packages\calibre\gui2\device.py", line 86, in run
  File "site-packages\calibre\gui2\device.py", line 490, in _books
  File "C:\Users\Rob\AppData\Local\Temp\calibre_7zr513\eopcl1_ios_local_db\Marvin_overlays.py", line 280, in books
  File "C:\Users\Rob\AppData\Local\Temp\calibre_7zr513\eopcl1_ios_local_db\Marvin_overlays.py", line 2302, in _restore_from_snapshot
  File "site-packages\calibre\devices\idevice\libimobiledevice.py", line 238, in copy_to_idevice
MemoryError

DEBUG: 65.5 MarvinManagerAction:shutting_down() DEBUG: 65.5 iOSReaderApp:shutdown() DEBUG: 65.5 iOSReaderApp:eject()

[GRiker]

Please try this:

• Disconnect your iPad
• Start calibre
• Open Preferences | Advanced | Plugins
• Click Show only user installed plugins (upper left corner)
• Open the Device Interface plugins section
• Double-click the iOSRA plugin
• In the General options tab, disable the Enabled booklist caching on device checkbox
• Click OK, exit calibre
• Reconnect your iPad, start Marvin and the calibre connector, try again.

Any different?

[Dhardrian]

Additional error detail from CMD window:

C:\Users\Rob>calibre-debug -g > debug_log.txt afc_receive_data: entire_len is larger than MAXIMUM_PACKET_SIZE, (384065 > 65536 )!

Traceback (most recent call last): File "site-packages\calibre\gui2\device.py", line 86, in run File "site-packages\calibre\gui2\device.py", line 490, in _books File "C:\Users\Rob\AppData\Local\Temp\calibre_tzbbkq\9a78xd_ios_local_db\Marvi n_overlays.py", line 280, in books File "C:\Users\Rob\AppData\Local\Temp\calibre_tzbbkq\9a78xd_ios_local_db\Marvi n_overlays.py", line 2302, in _restore_from_snapshot File "site-packages\calibre\devices\idevice\libimobiledevice.py", line 238, in copy_to_idevice MemoryError

[Dhardrian]

Greg,

Error does not occur with caching turned off. “Get list of books on device” completes in 20 sec. CMD window still displays the “warning”:

C:\Users\Rob>calibre-debug -g > debug_log.txt afc_receive_data: entire_len is larger than MAXIMUM_PACKET_SIZE, (384065 > 65536 )!

Rob

From: GRiker [mailto:notifications@github.com] Sent: Tuesday, April 29, 2014 4:55 PM To: GRiker/calibre-marvin-manager Cc: Robert Mohr Subject: Re: [calibre-marvin-manager] Get List of books on Device Fails (#46)

Please try this:

• Disconnect your iPad
• Start calibre
• Open Preferences | Advanced | Plugins
• Click Show only user installed plugins (upper left corner)
• Open the Device Interface plugins section
• Double-click the iOSRA plugin
• In the General options tab, disable the Enabled booklist caching on device checkbox
• Click OK, exit calibre
• Reconnect your iPad, start Marvin and the calibre connector, try again.

Any different?

— Reply to this email directly or view it on GitHubhttps://github.com/GRiker/calibre-marvin-manager/issues/46#issuecomment-41746383.

calibre 1.34 isfrozen: True is64bit: False Windows-7-6.1.7601-SP1 Windows ('32bit', 'WindowsPE') 32bit process running on 64bit windows ('Windows', '7', '6.1.7601') Python 2.7.5 Windows: ('7', '6.1.7601', 'SP1', 'Multiprocessor Free') DEBUG: 0.0 iOSReaderApp:initialize(v1.3.9) DEBUG: 0.0 iOSReaderApp:compile_ui() DEBUG: 0.1 iOSReaderApp:_init_prefs(prefs created under v1.3.8) DEBUG: 0.1 booklist_caching: True DEBUG: 0.1 cc_mappings: {u'My Library': {u'marvin_word_count': {u'field': u'#mm_word_count', u'combobox': u'Word count'}, u'marvin_collections': {u'field': u'#genre', u'combobox': u'Genre'}}} DEBUG: 0.1 debug_libimobiledevice: False DEBUG: 0.1 debug_plugin: True DEBUG: 0.1 development_mode: False DEBUG: 0.1 device_booklist_cache_limit: 5 DEBUG: 0.1 device_booklist_caching: False DEBUG: 0.1 kindle_enabled_formats: [u'MOBI'] DEBUG: 0.1 kindle_supported_formats: [u'MOBI'] DEBUG: 0.1 marvin_edit_collections_cb: True DEBUG: 0.1 marvin_protect_rb: False DEBUG: 0.1 marvin_replace_rb: False DEBUG: 0.1 marvin_update_rb: True DEBUG: 0.1 plugin_diagnostics: True DEBUG: 0.1 preferred_reader_app: u'Marvin' DEBUG: 0.1 iOSReaderApp:_get_connected_device_info() DEBUG: 4.4 mounting 'com.appstafarian.MarvinIP' DEBUG: 5.6 iOSReaderApp:_class_reconfigure('Marvin') DEBUG: 5.6 iOSReaderApp:_load_reader_app_overlays('Marvin') DEBUG: 5.6 loading 58 overlays DEBUG: 5.6 adding overlay '_compare_mainDb_profiles()' DEBUG: 5.6 adding overlay '_cover_subpath()' DEBUG: 5.6 adding overlay '_cover_to_thumb()' DEBUG: 5.6 adding overlay '_create_cover_element()' DEBUG: 5.6 adding overlay '_create_empty_booklist_db()' DEBUG: 5.6 adding overlay '_create_new_book()' DEBUG: 5.6 adding overlay '_dehydrate_booklist()' DEBUG: 5.6 adding overlay '_establish_local_booklist_db_path()' DEBUG: 5.6 adding overlay '_evaluate_original_cover()' DEBUG: 5.6 adding overlay '_evaluate_replaceable_cover()' DEBUG: 5.6 adding overlay '_get_field_items()' DEBUG: 5.6 adding overlay '_get_opf_tree()' DEBUG: 5.6 adding overlay '_get_opf_xml()' DEBUG: 5.6 adding overlay '_initialize_overlay()' DEBUG: 5.6 adding overlay '_localize_booklist_db()' DEBUG: 5.6 adding overlay '_localize_database_path()' DEBUG: 5.6 adding overlay '_parse_version()' DEBUG: 5.6 adding overlay '_parse_xml()' DEBUG: 5.6 adding overlay '_profile_db()' DEBUG: 5.6 adding overlay '_rehydrate_booklist()' DEBUG: 5.6 adding overlay '_remove_existing_copy()' DEBUG: 5.6 adding overlay '_report_upload_results()' DEBUG: 5.6 adding overlay '_reset_ios_connection()' DEBUG: 5.6 adding overlay '_restore_from_snapshot()' DEBUG: 5.6 adding overlay '_schedule_metadata_update()' DEBUG: 5.6 adding overlay '_snapshot_booklist()' DEBUG: 5.6 adding overlay '_stage_command_file()' DEBUG: 5.6 adding overlay '_update_epub_metadata()' DEBUG: 5.6 adding overlay '_validate_dehydrated_booklist()' DEBUG: 5.6 adding overlay '_wait_for_command_completion()' DEBUG: 5.6 adding overlay '_watchdog_timed_out()' DEBUG: 5.6 adding overlay '_xform_metadata_via_plugboard()' DEBUG: 5.6 adding overlay 'add_books_to_metadata()' DEBUG: 5.6 adding overlay 'books()' DEBUG: 5.6 adding overlay 'can_handle()' DEBUG: 5.6 adding overlay 'can_handle_windows()' DEBUG: 5.6 adding overlay 'delete_books()' DEBUG: 5.6 adding overlay 'eject()' DEBUG: 5.6 adding overlay 'from_json()' DEBUG: 5.6 adding overlay 'get_busy_flag()' DEBUG: 5.6 adding overlay 'get_cc_mapping()' DEBUG: 5.6 adding overlay 'get_file()' DEBUG: 5.6 adding overlay 'guess_type()' DEBUG: 5.6 adding overlay 'is_usb_connected()' DEBUG: 5.6 adding overlay 'is_usb_connected_windows()' DEBUG: 5.6 adding overlay 'post_yank_cleanup()' DEBUG: 5.6 adding overlay 'prepare_addable_books()' DEBUG: 5.6 adding overlay 'remove_books_from_metadata()' DEBUG: 5.6 adding overlay 'set_busy_flag()' DEBUG: 5.6 adding overlay 'set_cc_mapping()' DEBUG: 5.6 adding overlay 'shutdown()' DEBUG: 5.6 adding overlay 'sort_key()' DEBUG: 5.6 adding overlay 'startup()' DEBUG: 5.6 adding overlay 'sync_booklists()' DEBUG: 5.6 adding overlay 'thumbnail()' DEBUG: 5.6 adding overlay 'to_json()' DEBUG: 5.6 adding overlay 'upload_books()' DEBUG: 5.6 adding overlay 'xml_to_unicode()' DEBUG: 5.6 iOSReaderApp:_initialize_overlay(Marvin) DEBUG: 5.6 existing thumb cache at 'C:\Users\Rob\AppData\Local\calibre-cache\Marvin\thumbs.zip' DEBUG: 5.6 ~~~~~ switching to Marvin overlay ~~~~~~ Starting up... DEBUG: 15.8 iOSReaderApp:startup() DEBUG: 15.8 Waiting for calibre connector... DEBUG: 18.0 Marvin version: (2, 7, 0) DEBUG: 19.0 iOSReaderApp:reset() DEBUG: 19.0 iOSReaderApp:open() DEBUG: 19.0 Vendor ID (vid):05ac Product ID: (pid):12ab Started up in 30.15 seconds with 10930 books DEBUG: 37.3 iOSReaderApp:set_progress_reporter() Job: 1 Get device information started DEBUG: 37.3 iOSReaderApp:get_device_information() DEBUG: 37.3 DeviceClass : iPad DEBUG: 37.3 DeviceColor : black DEBUG: 37.3 DeviceName : Rob Mohr’s iPad DEBUG: 37.3 FSBlockSize : 4096 DEBUG: 37.3 FSFreeBytes : 78,808,989,696 DEBUG: 37.3 FSTotalBytes : 123,994,841,088 DEBUG: 37.3 FirmwareVersion : iBoot-1940.10.58 DEBUG: 37.3 HardwareModel : P101AP DEBUG: 37.3 ModelNumber : ME392 DEBUG: 37.3 PasswordProtected : False DEBUG: 37.3 ProductType : iPad3,4 DEBUG: 37.3 ProductVersion : 7.1.1 DEBUG: 37.3 SerialNumber : DMPK5B3TFCYF DEBUG: 37.3 TimeIntervalSince1970: 1398815911.35 DEBUG: 37.3 TimeZone : America/Los_Angeles DEBUG: 37.3 TimeZoneOffsetFromUTC: -25200.0 DEBUG: 37.3 UniqueDeviceID : f86c1193a1c728e07d3e88f6f80694e8d9224bd4 DEBUG: 37.3 iOSReaderApp:free_space() DeviceJob: 1 Get device information done, calling callback Job: 1 Get device information finished No details available. DeviceJob: 1 Get device information callback returned DEBUG: 38.1 iOSReaderApp:set_progress_reporter() Job: 2 Get list of books on device started DEBUG: 38.1 iOSReaderApp:books() DEBUG: 38.1 iOSReaderApp:_localize_database_path() Job: 1 Get device information finished No details available. DEBUG: 38.1 copying local_db from u'/Library/mainDb.sqlite' DEBUG: 39.1 iOSReaderApp:_restore_from_snapshot() DEBUG: 39.1 Analyzing local booklist DEBUG: 39.6 local booklist.db validated DEBUG: 56.1 iOSReaderApp:_rehydrate_booklist() DEBUG: 58.0 iOSReaderApp:_log_metrics(Marvin) DeviceJob: 2 Get list of books on device done, calling callback DeviceJob: metadata_downloaded: Starting set_books_in_library DEBUG: 58.5 PluginMetricsLogger:run(logged to 'iOS reader applications') DeviceJob: set_books_in_library: books to process= 10930 Job: 2 Get list of books on device finished No details available. DeviceJob: set_books_in_library finished: time= 0.174999952316 DeviceJob: metadata_downloaded: updating views DeviceJob: metadata_downloaded: syncing DEBUG: 60.2 iOSReaderApp:settings(format_map for 'Marvin': [u'epub']) DEBUG: 68.4 iOSReaderApp:settings(format_map for 'Marvin': [u'epub']) DeviceJob: metadata_downloaded: refreshing ondevice DeviceJob: metadata_downloaded: sending metadata_available signal DeviceJob: 2 Get list of books on device callback returned Job: 2 Get list of books on device finished No details available. DEBUG: 247.5 iOSReaderApp:shutdown() DEBUG: 247.5 iOSReaderApp:eject()

[GRiker]

Finally, some progress. I need to figure out why that memory error is occurring with device caching enabled. I put that switch in in response to this issue. If you are only using one iDevice, then leaving the switch off is preferable. I will probably ask you to try some more revisions in the next day or two to see if I can pin down what's causing the memory error.

Thanks for your patience.

[Dhardrian]

Greg, Repeated the deletion of the iOSRA booklist cache before running test. “Get list of books on device” finished in 2 min 52 sec. with no errors with caching turned off. Rob

From: GRiker [mailto:notifications@github.com] Sent: Tuesday, April 29, 2014 4:55 PM To: GRiker/calibre-marvin-manager Cc: Robert Mohr Subject: Re: [calibre-marvin-manager] Get List of books on Device Fails (#46)

Please try this:

• Disconnect your iPad
• Start calibre
• Open Preferences | Advanced | Plugins
• Click Show only user installed plugins (upper left corner)
• Open the Device Interface plugins section
• Double-click the iOSRA plugin
• In the General options tab, disable the Enabled booklist caching on device checkbox
• Click OK, exit calibre
• Reconnect your iPad, start Marvin and the calibre connector, try again.

Any different?

— Reply to this email directly or view it on GitHubhttps://github.com/GRiker/calibre-marvin-manager/issues/46#issuecomment-41746383.

[Dhardrian]

Greg,

Okay, sounds good. I am in town all this week. Back out traveling starting this coming Sunday.

Rob

From: GRiker [mailto:notifications@github.com] Sent: Tuesday, April 29, 2014 5:12 PM To: GRiker/calibre-marvin-manager Cc: Robert Mohr Subject: Re: [calibre-marvin-manager] Get List of books on Device Fails (#46)

Finally, some progress. I need to figure out why that memory error is occurring with device caching enabled. I put that switch in in response to this issue. If you are only using one iDevice, then leaving the switch off is preferable. I will probably ask you to try some more revisions in the next day or two to see if I can pin down what's causing the memory error.

Thanks for your patience.

— Reply to this email directly or view it on GitHubhttps://github.com/GRiker/calibre-marvin-manager/issues/46#issuecomment-41747477.

[GRiker]

I've updated iOSRA to handle one of the two memory errors reported. A fix for the other memory error requires a change to the main calibre program, so testing that will have to wait until after the next release of calibre.

In the meantime, please update to the latest iOSRA build available from this link.

Before installing the update:

• Please delete the existing iOSRA booklist cache using the Developer menu while Marvin is connected.
• Disconnect your iPad and install the update.
• Exit calibre, connect iPad.
• Start Marvin and calibre connector.
• Restart calibre. The cache will be rebuilt the first time, then operation should return to normal.

If you open the configuration panel, you will see a new switch in the General options tab, Enabled booklist caching on device. The purpose of this switch is to keep a copy of the booklist cache on your iDevice. If you are using only one iDevice that you're using with calibre and Marvin, this checkbox should be disabled. If you are using more than one iDevice, this checkbox should be enabled.

There is always a cache of the booklist for the most recently connected iDevice kept on the computer, so if you're only using one iDevice, it's actually slower to keep a copy on the iDevice, as the file needs to be copied to the iDevice every time it's updated. Particularly with large Marvin libraries, this can be slow, as the write speed for iDevices is quite slow (~6MB/second). For example, the cached booklist for my 4200 book test library is 112 MB.

If you're using more than one iDevice, there will still be some significant time savings with the switch enabled.

Here's what to expect with this update: After deleting the caches and doing the initial connection, load times should be back to where they were, but without this error with large libraries:

afc_receive_data: entire_len is larger than MAXIMUM_PACKET_SIZE, (384065 > 65536 )!

I believe I have a fix for the other memory error in place, but @Dhardrian, we'll have to wait to test that until the next release of calibre is available. In the meantime, keep the booklist caching switch disabled and you should be fine. @WyndhamMisio, you should enable that switch since you're using two devices, and your libraries aren't large enough to trigger the memory error that @Dhardrian is experiencing.

[GRiker]

@Dhardrian, Version 1.3.7 of calibre is now available. With the latest prerelease build of iOSRA, you should be able to send thousands of books at a time to Marvin. Please confirm, and then close this ticket if that particular problem is resolved.

[Dhardrian]

Greg,

I am testing the new release of Calibre 1.3.7 as well as the pre-releases of the iOSRA and MarvinXD plugins against the full library (10K+). I will send you a message letting you know how it went after it finishes. I am off to Europe in the morning so I will be unable to perform any further testing until I return next Saturday the 24th if there remain issues.

Rob

From: GRiker [mailto:notifications@github.com] Sent: Friday, May 16, 2014 2:13 AM To: GRiker/calibre-marvin-manager Cc: Robert Mohr Subject: Re: [calibre-marvin-manager] Get List of books on Device Fails (#46)

@Dhardrianhttps://github.com/Dhardrian, Version 1.3.7 of calibre is now available. With the latest prerelease build of iOSRAhttps://www.dropbox.com/s/vaihbwks17sfdz4/iOS%20reader%20applications.zip, you should be able to send thousands of books at a time to Marvin. Please confirm, and then close this ticket if that particular problem is resolved.

— Reply to this email directly or view it on GitHubhttps://github.com/GRiker/calibre-marvin-manager/issues/46#issuecomment-43311975.

[Dhardrian]

Greg,

It completed successfully with an elapsed time reported in Calibre jobs of 113m 22s for the upload of 10930 books, and 20s for the metadata update. Actual elapsed time was closer to 2 hours and 20 minutes, which includes the processing time before the job started.

Thanks for the work to get this working, it is greatly appreciated!

Rob

From: Robert Mohr Sent: Friday, May 16, 2014 10:32 PM To: 'GRiker/calibre-marvin-manager'; GRiker/calibre-marvin-manager Subject: RE: [calibre-marvin-manager] Get List of books on Device Fails (#46)

Greg,

I am testing the new release of Calibre 1.3.7 as well as the pre-releases of the iOSRA and MarvinXD plugins against the full library (10K+). I will send you a message letting you know how it went after it finishes. I am off to Europe in the morning so I will be unable to perform any further testing until I return next Saturday the 24th if there remain issues.

Rob

From: GRiker [mailto:notifications@github.com] Sent: Friday, May 16, 2014 2:13 AM To: GRiker/calibre-marvin-manager Cc: Robert Mohr Subject: Re: [calibre-marvin-manager] Get List of books on Device Fails (#46)

@Dhardrianhttps://github.com/Dhardrian, Version 1.3.7 of calibre is now available. With the latest prerelease build of iOSRAhttps://www.dropbox.com/s/vaihbwks17sfdz4/iOS%20reader%20applications.zip, you should be able to send thousands of books at a time to Marvin. Please confirm, and then close this ticket if that particular problem is resolved.

— Reply to this email directly or view it on GitHubhttps://github.com/GRiker/calibre-marvin-manager/issues/46#issuecomment-43311975.

[GRiker]

Thanks for the followup I appreciate your patience while resolving thus problem. Travel safely.