Send to Device fails with memory error

Dhardrian commented 10 years ago

The "Send to device" command is intermittently failing. I am updating meta data on existing books on the iDevice (i.e. Marvin configured to update rather than do nothing on existing). I have been able to trigger this error by sending as few as 901 books. Sending an entire library of 10930 books fails on every attempt.

Marvin version: 2.7 iDevice: iPad Model ME392LL/A Calibre version: 1.34.0 Marvin XD plugin version: iOS reading applications plugin version: Operating system: Windows 7, 64-bit iTunes version: 11.1.5.5

The specific steps to recreate the problem: Attach iPad: Run Marvin: Select Get Books: Select Calibre In Calibre, select "Send to Device"

Description of the problem: Send to device fails with memory error.

Dhardrian commented 10 years ago

Forking issue to IOSRA as requested

Dhardrian commented 10 years ago

Attempt to send 10K+ resulted in lockup. CMD traceback:

C:\Users\Rob>calibre-debug -g > debug_log.txt Exception ValueError: 'I/O operation on closed file' in <bound method ZipFile. del of <calibre.utils.zipfile.ZipFile instance at 0x48FB64B8>> ignored Exception ValueError: 'I/O operation on closed file' in <bound method ZipFile. del of <calibre.utils.zipfile.ZipFile instance at 0x25DAD9E0>> ignored Exception ValueError: 'I/O operation on closed file' in <bound method ZipFile. del of <calibre.utils.zipfile.ZipFile instance at 0x499D20A8>> ignored Exception in thread Thread-5: Traceback (most recent call last): File "threading.py", line 808, in bootstrap_inner File "site-packages\calibre\gui2\device.py", line 409, in run File "site-packages\calibre\gui2\device.py", line 98, in run File "site-packages\calibre\gui2\device.py", line 68, in job_done File "site-packages\calibre__init.py", line 202, in prints MemoryError

C++ exception thrown in slot: bad allocation Receiver name: "qt_scrollarea_viewport" Receiver class: QWidget Event type: 12

GRiker commented 10 years ago

I've built a version of iOSRA that should make some progress towards supporting very large downloads.

Please download and install from this link.

The change I made is to break up large 'Send to device' requests into chunks. By default, I've set the chunk size to 100 books. The chunk size shouldn't impact performance significantly, but if this works we can experiment with making it bigger.

I'd like to ask that you test this in steps:

Try sending 100 books whose metadata will be updated (in other words, the books are already in Marvin, and you have selected 'Update metadata' in Marvin options. Note the Running time in the Jobs window after this completes.
Try sending 500 books whose metadata will be updated, note the Running time after the job completes.
Try sending 2000 books, note running time.

If this works, let's move on to the next test:

Try sending your entire library. Note the running time.

It's entirely possible that these tests will still fail, so it would be helpful to run in debug mode and capture the debug log. You can email it to me at the address I previously provided.

I'm going to close the other issue you opened in the MXD issue tracker. The rule of thumb for deciding where to open an issue is "If the bug occurs while the MXD window is open, create the issue in the MXD bug tracker. Otherwise, create the issue in the iOSRA tracker".

Dhardrian commented 10 years ago

100 book test - success Upload 100 books to device – 2m 9s Send metadata to device – 25s

500 book test - success Upload 500 books to device – 5m 37s Send metadata to device – 32s

2000 book test - success Upload 2000 books to device – 20m 3s Send metadata to device – 1m 1s

10930 book test - failed Upload 10930 books to device – failure at 79m 59s

Debug log being sent via secondary address

Additional details: Exception in thread Thread-5: Traceback (most recent call last): File "threading.py", line 808, in bootstrap_inner File "site-packages\calibre\gui2\device.py", line 399, in run File "site-packages\calibre\gui2\device.py", line 271, in detect_device File "site-packages\calibre\devices\scanner.py", line 330, in is_device_connec ted File "C:\Users\Rob\AppData\Local\Temp\calibre_za71sr\zhterx_ios_local_db\Marvi n_overlays.py", line 743, in is_usb_connected File "calibre_plugins.ios_reader_apps.__init", line 145, in __log_location AttributeError: 'NoneType' object has no attribute '_getframe'

Traceback (most recent call last): File "site-packages\calibre\gui2\device.py", line 86, in run File "site-packages\calibre\gui2\device.py", line 571, in _upload_books File "C:\Users\Rob\AppData\Local\Temp\calibre_za71sr\zhterx_ios_local_db\Marvi n_overlays.py", line 1802, in upload_books File "C:\Users\Rob\AppData\Local\Temp\calibre_za71sr\zhterx_ios_local_db\Marvi n_overlays.py", line 1752, in _upload_subset File "C:\Users\Rob\AppData\Local\Temp\calibre_za71sr\zhterx_ios_local_db\Marvi n_overlays.py", line 2937, in _stage_command_file File "site-packages\calibre\devices\idevice\libimobiledevice.py", line 656, in write File "site-packages\calibre\devices\idevice\libimobiledevice.py", line 954, in _afc_file_write MemoryError

C:\Users\Rob>calibre-debug -g > debug_log_10930.txt Exception ValueError: 'I/O operation on closed file' in <bound method ZipFile. del of <calibre.utils.zipfile.ZipFile instance at 0x36289670>> ignored Exception ValueError: 'I/O operation on closed file' in <bound method ZipFile. del of <calibre.utils.zipfile.ZipFile instance at 0x36FA0850>> ignored Exception ValueError: 'I/O operation on closed file' in <bound method ZipFile. del of <calibre.utils.zipfile.ZipFile instance at 0x28B4D6C0>> ignored

GRiker commented 10 years ago

Please download and install an updated version from this link. This should get through the full set of books. It still may crash later in the process when trying to report the results.

GRiker commented 10 years ago

I've created a new issue for iOSRA beta tester #80. Please go to that issue, then click the Subscribe button in the Notifications area. That will make it easier for me to notify all of the beta testers when a new version is available.

GRiker commented 10 years ago

@Dhardrian, I've found another bug related to multiple file transfers. This will require another code change within calibre proper, not the plugin. Unfortunately, that means waiting another release cycle of calibre to get the patch in place. Sorry.

Dhardrian commented 10 years ago

Let me know if/when I can help

Sent from my iPad

On May 10, 2014, at 3:14 PM, "GRiker" notifications@github.com<mailto:notifications@github.com> wrote:

@Dhardrianhttps://github.com/Dhardrian, I've found another bug related to multiple file transfers. This will require another code change within calibre proper, not the plugin. Unfortunately, that means waiting another release cycle of calibre to get the patch in place. Sorry.

— Reply to this email directly or view it on GitHubhttps://github.com/GRiker/calibre-ios-reader-applications/issues/79#issuecomment-42756008.

GRiker commented 10 years ago

Thanks, and sorry for the delay. I've already submitted the patch to Kovid for the next release, which should be next Friday. I will ping you when it's available.

Are you by any chance running calibre from source? If so, you could test the change immediately. But if you're not, it's probably easier to just wait for the next release.

Dhardrian commented 10 years ago

Greg,

I am not running from source. Timing is good, I am headed to Corpus Crisiti tomorrow, but will be back before Friday. Please send information about running from source if you have it. I could setup to do that later this week.

Rob

Sent from my iPad

On May 11, 2014, at 11:05 AM, "GRiker" notifications@github.com<mailto:notifications@github.com> wrote:

Thanks, and sorry for the delay. I've already submitted the patch to Kovid for the next release, which should be next Friday. I will ping you when it's available.

Are you by any chance running calibre from source? If so, you could test the change immediately. But if you're not, it's probably easier to just wait for the next release.

— Reply to this email directly or view it on GitHubhttps://github.com/GRiker/calibre-ios-reader-applications/issues/79#issuecomment-42778111.

GRiker commented 10 years ago

You can read about setting up a calibre development environment at this link. If you decide to do so, let me know after you get it running, and then I'll tell you what to do next.

Dhardrian commented 10 years ago

Re-opening issue. Problems have recurred in newer versions of Calibre and plugin

Philantrop / calibre-ios-reader-applications

Send to Device fails with memory error #79