Calibre with DeDRM plugin used to work, now breaks erratically (Feb 2019)

[stephenfpersonal]

I have been using Calibre and DeDRM for years to archive PURCHASED Amazon books and to convert them to EPUB for use on my Kobo. This week, it stopped working on certain files. Specifically, I purchased "Texas" by James Michener, "Storm Front" and "Fool Moon" by Jim Butcher, and "Downbelow Station" by CJ Cherryh. In all cases, Calibre throws the same error you would get if DeDRM was not installed. Surprisingly, "Panacea" (F. Paul Wilson) and "A Gentleman in Moscow" (Amor Towles) purchased during the same week were converted to EPUBs just fine.

Running Calibre 3.38.1, DeDRM 6.6.1, macOS 10.13.6. Debug log below if it's useful to anyone. No configuration changes between the last time I used Calibre three weeks ago, and the failures I am experiencing now. Any insight or suggestions? Thanks!

2019-02-08 15:50:35.397 Calibre[99757:8080424] deliverNotificationWithTitle()
* Notification delivered.
Job: 1 Convert book 1 of 1 (Texas) finished
Convert book 1 of 1 (Texas)
    Conversion options changed from defaults:
      minimum_line_height: 150.0
      italicize_common_cases: False
      epub_toc_at_end: True
      cover: u'/var/folders/_9/yk_kk7k17cs3nbd84kxzfxyh0000gn/C/calibre_3.38.1_tmp_iVqrJF/aJq5fk.jpeg'
      insert_blank_line: True
      extra_css: u'p {text-indent: 5em;}\nbody {orphans: 0; widows: 0;}'
      remove_paragraph_spacing_indent_size: 2.5
      insert_blank_line_size: 0.7
      delete_blank_paragraphs: False
      filter_css: u',font-family'
      format_scene_breaks: False
      preserve_cover_aspect_ratio: True
      verbose: 2
      unwrap_lines: False
      dehyphenate: False
      renumber_headings: False
      base_font_size: 12.0
      read_metadata_from_opf: u'/var/folders/_9/yk_kk7k17cs3nbd84kxzfxyh0000gn/C/calibre_3.38.1_tmp_iVqrJF/WBsS0E.opf'
      change_justification: u'left'
      enable_heuristics: True
      epub_inline_toc: True
      markup_chapter_headings: False
    Resolved conversion options
    calibre version: 3.38.1
    {'asciiize': False,
     'author_sort': None,
     'authors': None,
     'base_font_size': 12.0,
     'book_producer': None,
     'change_justification': u'left',
     'chapter': u"//*[((name()='h1' or name()='h2') and re:test(., '\\s*((chapter|book|section|part)\\s+)|((prolog|prologue|epilogue)(\\s+|$))', 'i')) or @class = 'chapter']",
     'chapter_mark': u'pagebreak',
     'comments': None,
     'cover': u'/var/folders/_9/yk_kk7k17cs3nbd84kxzfxyh0000gn/C/calibre_3.38.1_tmp_iVqrJF/aJq5fk.jpeg',
     'debug_pipeline': None,
     'dehyphenate': False,
     'delete_blank_paragraphs': False,
     'disable_font_rescaling': False,
     'dont_split_on_page_breaks': False,
     'duplicate_links_in_toc': False,
     'embed_all_fonts': False,
     'embed_font_family': None,
     'enable_heuristics': True,
     'epub_flatten': False,
     'epub_inline_toc': True,
     'epub_toc_at_end': True,
     'epub_version': u'2',
     'expand_css': False,
     'extra_css': u'p {text-indent: 5em;}\nbody {orphans: 0; widows: 0;}',
     'extract_to': None,
     'filter_css': u',font-family',
     'fix_indents': True,
     'flow_size': 260,
     'font_size_mapping': None,
     'format_scene_breaks': False,
     'html_unwrap_factor': 0.4,
     'input_encoding': None,
     'input_profile': <calibre.customize.profiles.InputProfile object at 0x11280b590>,
     'insert_blank_line': True,
     'insert_blank_line_size': 0.7,
     'insert_metadata': False,
     'isbn': None,
     'italicize_common_cases': False,
     'keep_ligatures': False,
     'language': None,
     'level1_toc': None,
     'level2_toc': None,
     'level3_toc': None,
     'line_height': 0.0,
     'linearize_tables': False,
     'margin_bottom': 5.0,
     'margin_left': 5.0,
     'margin_right': 5.0,
     'margin_top': 5.0,
     'markup_chapter_headings': False,
     'max_toc_links': 50,
     'minimum_line_height': 150.0,
     'no_chapters_in_toc': False,
     'no_default_epub_cover': False,
     'no_inline_navbars': False,
     'no_svg_cover': False,
     'output_profile': <calibre.customize.profiles.OutputProfile object at 0x11280b950>,
     'page_breaks_before': u"//*[name()='h1' or name()='h2']",
     'prefer_metadata_cover': False,
     'preserve_cover_aspect_ratio': True,
     'pretty_print': True,
     'pubdate': None,
     'publisher': None,
     'rating': None,
     'read_metadata_from_opf': u'/var/folders/_9/yk_kk7k17cs3nbd84kxzfxyh0000gn/C/calibre_3.38.1_tmp_iVqrJF/WBsS0E.opf',
     'remove_fake_margins': True,
     'remove_first_image': False,
     'remove_paragraph_spacing': False,
     'remove_paragraph_spacing_indent_size': 2.5,
     'renumber_headings': False,
     'replace_scene_breaks': u'',
     'search_replace': '[]',
     'series': None,
     'series_index': None,
     'smarten_punctuation': False,
     'sr1_replace': None,
     'sr1_search': None,
     'sr2_replace': None,
     'sr2_search': None,
     'sr3_replace': None,
     'sr3_search': None,
     'start_reading_at': None,
     'subset_embedded_fonts': False,
     'tags': None,
     'timestamp': None,
     'title': None,
     'title_sort': None,
     'toc_filter': None,
     'toc_threshold': 6,
     'toc_title': None,
     'transform_css_rules': '[]',
     'unsmarten_punctuation': False,
     'unwrap_lines': False,
     'use_auto_toc': False,
     'verbose': 2}
    Python function terminated unexpectedly: Texas
    InputFormatPlugin: MOBI Input running
    on /var/folders/_9/yk_kk7k17cs3nbd84kxzfxyh0000gn/C/calibre_3.38.1_tmp_iVqrJF/yJXfr1.azw3
    Found KF8 MOBI of type 'standalone'
    Traceback (most recent call last):
      File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 154, in main
        return run_entry_point()
      File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 114, in run_entry_point
        return getattr(pmod, func)()
      File "site-packages/calibre/utils/ipc/worker.py", line 199, in main
      File "site-packages/calibre/gui2/convert/gui_conversion.py", line 42, in gui_convert_override
      File "site-packages/calibre/gui2/convert/gui_conversion.py", line 27, in gui_convert
      File "site-packages/calibre/ebooks/conversion/plumber.py", line 1106, in run
      File "site-packages/calibre/customize/conversion.py", line 244, in __call__
      File "site-packages/calibre/ebooks/conversion/plugins/mobi_input.py", line 45, in convert
      File "site-packages/calibre/ebooks/mobi/reader/mobi8.py", line 86, in __call__
      File "site-packages/calibre/ebooks/mobi/reader/mobi6.py", line 157, in check_for_drm
    calibre.ebooks.DRMError: Texas

[ElleKayEm]

DRM removal happens when you first add the book to calibre, not during conversion. Here's how to get the log we need to see:

https://github.com/apprenticeharper/DeDRM_tools/blob/master/FAQs.md#i-cannot-solve-my-problem-with-the-dedrm-plugin-and-now-i-need-to-post-a-log-how-do-i-do-that

[stephenfpersonal]

My apologies. Here's the import log:

[6.17] Started up in 6.17 seconds with 3105 books
objc[14804]: Class FIFinderSyncExtensionHost is implemented in both /System/Library/PrivateFrameworks/FinderKit.framework/Versions/A/FinderKit (0x7fff8d860cd0) and /System/Library/PrivateFrameworks/FileProvider.framework/OverrideBundles/FinderSyncCollaborationFileProviderOverride.bundle/Contents/MacOS/FinderSyncCollaborationFileProviderOverride (0x11bd44cd8). One of the two will be used. Which one is undefined.
DeDRM v6.6.1: Trying to decrypt Texas.azw3
Using Library AlfCrypto DLL/DYLIB/SO
MobiDeDrm v0.42.
Copyright © 2008-2017 The Dark Reverser, Apprentice Harper et al.
Decrypting Kindle Format 8 ebook: Texas
Found 8 keys to try after 0.1 seconds
Crypto Type is: 2
DeDRM v6.6.1: Failed to decrypt with error: No key found in 8 keys tried.
DeDRM v6.6.1: Looking for new default Kindle Key after 0.1 seconds
Found k4Mac kinf2011 file: /Users/stephen/Library/Application Support/Kindle/storage/.kinf2011
Found k4Mac kindle-info file: /Users/stephen/Library/Application Support/Amazon/Kindle for Mac/storage/.kindle-info
Decrypted key file using IDString '6d8fb113e7e3' and UserName 'stephen'
Couldn't decrypt file.
DeDRM v6.6.1: Found 1 new key
MobiDeDrm v0.42.
Copyright © 2008-2017 The Dark Reverser, Apprentice Harper et al.
Decrypting Kindle Format 8 ebook: Texas
Got DSN key from database default_key_1
Found 4 keys to try after 1.1 seconds
Crypto Type is: 2
DeDRM v6.6.1: Ultimately failed to decrypt after 1.1 seconds. Read the FAQs at Harper's repository: https://github.com/apprenticeharper/DeDRM_tools/blob/master/FAQs.md
Running file type plugin DeDRM failed with traceback:
Traceback (most recent call last):
  File "site-packages/calibre/customize/ui.py", line 172, in _run_filetype_plugins
  File "calibre_plugins.dedrm.__init__", line 621, in run
  File "calibre_plugins.dedrm.__init__", line 571, in KindleMobiDecrypt
DeDRMError: DeDRM v6.6.1: Ultimately failed to decrypt after 1.1 seconds. Read the FAQs at Harper's repository: https://github.com/apprenticeharper/DeDRM_tools/blob/master/FAQs.md
Added Texas to db in: 0.4
Added 1 books in 2.4 seconds

Thanks for the pointer. I'm not new to Calibre, but I'm new to debug mode!

[ElleKayEm]

Where is the book from? Did you get it from the website for "Download & transfer via USB"? If so, you need to choose an E-ink Kindle (not a Fire) and enter its serial number into the DeDRM customization.

[tamarshallme]

@ElleKayEm so you're saying there's no way to DeDRM if you only own a Kindle Fire?

[ElleKayEm]

@tamarshallme You can use Kindle for PC/Mac version 1.24 or earlier installed on the same machine and user as calibre with DeDRM. (No customization of the plugin needed.)

[stephenfpersonal]

Where is the book from? Did you get it from the website for "Download & transfer via USB"? If so,

Purchased from Amazon, downloaded as specified. I've done this many times before successfully.

you need to choose an E-ink Kindle (not a Fire) and enter its serial number into the DeDRM customization.

Yes, and I double-checked the serial numbers. (I have four e-ink Kindles). Again, this has worked hundreds of times before. Something changed in January.

[ElleKayEm]

Try downloading the books again. If the serial number is entered correctly, it should work.

Or you can use Kindle for Mac 1.23 or earlier.

[stephenfpersonal]

Thanks. I deleted and re-downloaded them into Calibre twice. No success. Other books purchased today DeDRMed successfully using the same process with the same serial number(s).

I was able to extract them with Kindle for Mac 1.17, but it makes me nervous that something changed on Amazon's end, and I'm not sure what.

[ElleKayEm]

Choosing the same Kindle as the books that work? That is strange. If one of your Kindles is a 1st gen Paperwhite or earlier, you could also try importing a book directly from the device.

[Zazcallabah]

I have also run up against a similar error message

DeDRM v6.6.0: Processing: B007MB58WU_EBOK.azw...
MobiDeDrm v0.42.
Copyright © 2008-2017 The Dark Reverser, Apprentice Harper et al.
Decrypting Kindle Format 8 ebook: Fear and Loathing on the Campaign Trail '72
Got DSN key from database C:\Users\Zaz\AppData\Local\DeDRMPrefs\kindlekey.k4i
Found 4 keys to try after 0.0 seconds
Crypto Type is: 2
Error decrypting book after 0.0 seconds: No key found in 4 keys tried.
Traceback (most recent call last):
  File "C:\src\eBookConverter\DeDRM_tools-master\DeDRM_Windows_Application\DeDRM_App\DeDRM_lib\lib\k4mobidedrm.py", line 259, in decryptBook
    book = GetDecryptedBook(infile, kDatabases, androidFiles, serials, pids, starttime)
  File "C:\src\eBookConverter\DeDRM_tools-master\DeDRM_Windows_Application\DeDRM_App\DeDRM_lib\lib\k4mobidedrm.py", line 233, in GetDecryptedBook
    mb.processBook(totalpids)
  File "C:\src\eBookConverter\DeDRM_tools-master\DeDRM_Windows_Application\DeDRM_App\DeDRM_lib\lib\mobidedrm.py", line 471, in processBook
    raise DrmException(u"No key found in {0:d} keys tried.".format(len(goodpids)))
DrmException: No key found in 4 keys tried.
DeDRM v6.6.0: Failed

DeDRM v6.6.0: Complete:  Successes: 0, Failures: 1

Book was purchased today, and downloaded using Kindleforpc 1.24.3 with the rendertest file removed. This has worked earlier, and I just tested and it still works on my old books, if i delete them and download them again through kindleforpc.

[ElleKayEm]

Books are from the same installation of Kindle for PC? Can be both be opened and read in Kindle for PC? And both Kindle for PC where the books were downloaded are on the same machine and user as DeDRM?

I notice you're not using the latest version of DeDRM, but that shouldn't matter here.

[Zazcallabah]

Yes the latest version broke my scripts for some reason. Something about python references. See https://github.com/apprenticeharper/DeDRM_tools/issues/650

I can reproduce the error by doing the following: Delete the book from my kindleforpc device. Download it again. Download any other book I have to compare it with. Run my script. (basically "python DeDRM_App.pyw <path to azw>") The other book (bought 2019-01-17) will convert as it should, but "Fear and Loathing" (bought 2019-02-11) wont.

[ElleKayEm]

Hmm, usually it's all or nothing with a particular installation of a Kindle app or device...unless the book is a rental/loan or one of the included dictionaries.

Have you tried the calibre plugin?

[Zazcallabah]

No, but would it really find more keys to try?

[ElleKayEm]

Well, it should be the same. But sometimes people have better luck with the plugin.

[Zazcallabah]

Update: I had better luck by trying another computer. I installed kindleforpc and calibre and dedrm on my work laptop, and here both my script and the calibre plugin works. (I tested both)

Is there a way to compare the k4i files to figure out what went wrong on the other computer?

[Zazcallabah]

Is there a way to force kindle to regenerate the k4i file without reinstalling the application or re-registering?

[ElleKayEm]

Keys are definitely going to be different on another computer. Each app installation/device has its own key.

Sounds like the key changed for that installation of the app and DeDRM isn't picking up the new one. I'd try completely uninstalling and reinstalling Kindle for PC. There are instructions here:

https://github.com/apprenticeharper/DeDRM_tools/blob/master/FAQs.md#i-have-installed-the-calibre-plugin-and-the-book-is-not-already-in-calibre-but-the-drm-does-not-get-removed-it-is-a-kindle-book

(They advise using 1.17 or earlier, but as you know, you can now use 1.24 or earlier.)

[Zazcallabah]

Actually, I just deleted the dedrm cached kindlekey.k4i from %LocalAppData%\DeDrmPrefs and that solved the issue. :D

Thanks for all the help.

[ElleKayEm]

Oh, that's a lot easier! Thanks for reporting what worked.

[ElleKayEm]

These tools are intended for books you own.

[ElleKayEm]

I want the digital version.

Then buy it.

[ElleKayEm]

The developers of these tools have always made their policy on this very clear.

https://github.com/apprenticeharper/DeDRM_tools/blob/master/FAQs.md#i-cant-get-the-tools-to-work-on-my-rented-or-library-ebooks

https://github.com/apprenticeharper/DeDRM_tools/blob/master/FAQs.md#the-drm-wasnt-removed-and-the-log-says-failed-to-decrypt-with-error-cannot-decode-library-or-rented-ebooks-what-now

[ElleKayEm]

Not going to argue. Just pointing out the policy here.