Big Sur Kindle Does not decrypt

lauraannwilliams commented 3 years ago

CheckList

[x] The Title and The Log Title are setted correctly.
[x] Clarified about my environment.
[x] Code block is used for the log.

Title

My Environment

Calibre: `5.7.2`

Kindle: `1.26.1`

DeDRM: `7.0.0b4`

Log

I have a brand new Macbook running Big Sur. My old harddrive died so everything is all new installs and trying to get set up again.

I have installed and Registered Kindle on this machine, and downloaded (but not opened) a file. Calibre is running the same machine/user I have run the chmod command to block the new file format The files I'm downloading are named like B00AP949RQ_EBOK.azw but calibre is flagging them as 'KFX-ZIP' files

Note: this issue isn't happening on every file, but I don't know if that means some are drm'd and some are not, or if some are working and some are not. I can provide a log file with a file that does work if that would be helpful.

Log Title

[tmp4fbezz4d.txt](https://github.com/apprenticeharper/DeDRM_tools/files/5720615/tmp4fbezz4d.txt) ```log PUT YOUR LOG ```

lauraannwilliams commented 3 years ago

Thank you for your time and work, btw

ElleKayEm commented 3 years ago

You are still seeing KFX-ZIP after running the chmod command and then re-downloading and re-adding your books to calibre? If so, it would seem that Kindle for Mac is ignoring the fact that it can't run renderer-test and is downloading KFX format anyway. (Kindle for Mac gives the .azw extension to all of the main book files regardless of the underlying format. If you also see .voucher/.res/.md files in the same folder, that's KFX format. KFX-ZIP is what happens when the KFX Input plugin has gathered the multiple files together, but DeDRM wasn't able to decrypt it.)

If you can get non-KFX format, you should be able to remove the DRM with your current calibre/DeDRM versions. If you are unable to prevent KFX format downloads and Kindle for Mac has not auto-updated past 1.26, then DeDRM 6.8.1 loaded into calibre 4.23 will work.

ZolaLa9 commented 3 years ago

Did you try sudo chmod -x as well as chmod -x? Sometimes chmod -x doesn't work and you need to add sudo first (it will ask for an admin password, which you will need to provide).

Also, make sure the filepath is correct. If K4Mac isn't in your Applications folder, you will need to revise the filepath accordingly.

Finally, it's weird if it's working for some books and not others. chmod -x will block KFX. I think it either works or it doesn't work. It doesn't work on some books, but not others, which suggests something else may be going on ...

ElleKayEm commented 3 years ago

What are the extensions in calibre for the files that worked?

lauraannwilliams commented 3 years ago

I've double checked the chmod and there was an error there - files are being flagged as AZW files now. However, they're still not decrypting.

The ones that ARE decrypted are all flagged as AZW3 files

Here's a log with one successful and one failed file.

tmptqjfcgw4.txt

Thanks to everyone responding

ElleKayEm commented 3 years ago

The book that worked was already DRM-free to begin with. A key is found and decrypted, but it is not unlocking the DRM book. The Kindle for Mac installation that you are using to download these is the one that is currently on the same machine and user as calibre with DeDRM?

lauraannwilliams commented 3 years ago

yes its all the same machine. i'll see if i can figure out if any of the other books that opened had drm

ElleKayEm commented 3 years ago

It should be all or nothing. If key and book are from the same installation of Kindle for Mac, they should work. I don't know if the problem here is beta DeDRM 7, something with Big Sur, something about your setup, or some combination of those. I have seen reports of things working on Big Sur with DeDRM 6.8.1 installed in calibre 4.23.

lauraannwilliams commented 3 years ago

In an effort to provide more information, I downloaded and tried (and failed) to decrypt with the following kindle versions

1.26.1 1.27 1.28 1.29 1.30

My Calibre, OS and DeDRM version all stayed the same. Each time I deleted both the app and the application support folder, and reran the chmod hack.

Off into work time tomorrow so not sure when I'll be able to try again

One of many job logs

```log Convert book 1 of 1 (Proto Zoa) DeDRM v7.0.0: Trying to decrypt o6q8nezt.azw Using Library AlfCrypto Python Using Library AlfCrypto Python MobiDeDrm v1.0. Copyright © 2008-2020 The Dark Reverser, Apprentice Harper et al. Decrypting Kindle Format 8 ebook: Proto Zoa Got SerialNumber from database default_key Got UsernameHash from database default_key Found 4 keys to try after 0.1 seconds Crypto Type is: 2 DeDRM v7.0.0: Failed to decrypt with error: No key found in 4 keys tried. DeDRM v7.0.0: Looking for new default Kindle Key after 0.1 seconds Found k4Mac kinf2018 file: /Users/pengy/Library/Application Support/Kindle/storage/.kinf2018 trying username b'pengy' on file /Users/pengy/Library/Application Support/Kindle/storage/.kinf2018 trying IDString: b'b1d87f206bbd' Decrypted key file using IDString 'b1d87f206bbd' and UserName 'pengy' DeDRM v7.0.0: Found 1 new key MobiDeDrm v1.0. Copyright © 2008-2020 The Dark Reverser, Apprentice Harper et al. Decrypting Kindle Format 8 ebook: Proto Zoa Got SerialNumber from database default_key_1 Got UsernameHash from database default_key_1 Found 4 keys to try after 0.6 seconds Crypto Type is: 2 DeDRM v7.0.0: Ultimately failed to decrypt after 0.6 seconds. Read the FAQs at Harper's repository: https://github.com/apprenticeharper/DeDRM_tools/blob/master/FAQs.md Conversion options changed from defaults: filter_css: 'background,,color,background-color' disable_font_rescaling: True output_profile: 'tablet' unsmarten_punctuation: True cover: '/var/folders/pq/6qqzm60d6pg555vgfqns87400000gn/C/calibre_5.7.2_tmp_ops5_zhr/c_iot7gw.jpeg' verbose: 2 read_metadata_from_opf: '/var/folders/pq/6qqzm60d6pg555vgfqns87400000gn/C/calibre_5.7.2_tmp_ops5_zhr/5zmn73yr.opf' Resolved conversion options calibre version: 5.7.2 {'asciiize': False, 'author_sort': None, 'authors': None, 'base_font_size': 0.0, 'book_producer': None, 'change_justification': 'original', 'chapter': "//*[((name()='h1' or name()='h2') and re:test(., " "'\\s*((chapter|book|section|part)\\s+)|((prolog|prologue|epilogue)(\\s+|$))', " "'i')) or @class = 'chapter']", 'chapter_mark': 'pagebreak', 'comments': None, 'cover': '/var/folders/pq/6qqzm60d6pg555vgfqns87400000gn/C/calibre_5.7.2_tmp_ops5_zhr/c_iot7gw.jpeg', 'debug_pipeline': None, 'dehyphenate': True, 'delete_blank_paragraphs': True, 'disable_font_rescaling': True, 'dont_split_on_page_breaks': False, 'duplicate_links_in_toc': False, 'embed_all_fonts': False, 'embed_font_family': None, 'enable_heuristics': False, 'epub_flatten': False, 'epub_inline_toc': False, 'epub_toc_at_end': False, 'epub_version': '2', 'expand_css': False, 'extra_css': None, 'extract_to': None, 'filter_css': 'background,,color,background-color', 'fix_indents': True, 'flow_size': 260, 'font_size_mapping': None, 'format_scene_breaks': True, 'html_unwrap_factor': 0.4, 'input_encoding': None, 'input_profile': , 'insert_blank_line': False, 'insert_blank_line_size': 0.5, 'insert_metadata': False, 'isbn': None, 'italicize_common_cases': True, 'keep_ligatures': False, 'language': None, 'level1_toc': None, 'level2_toc': None, 'level3_toc': None, 'line_height': 0.0, 'linearize_tables': False, 'margin_bottom': 5.0, 'margin_left': 5.0, 'margin_right': 5.0, 'margin_top': 5.0, 'markup_chapter_headings': True, 'max_toc_links': 50, 'minimum_line_height': 120.0, 'no_chapters_in_toc': False, 'no_default_epub_cover': False, 'no_inline_navbars': False, 'no_svg_cover': False, 'output_profile': , 'page_breaks_before': "//*[name()='h1' or name()='h2']", 'prefer_metadata_cover': False, 'preserve_cover_aspect_ratio': False, 'pretty_print': True, 'pubdate': None, 'publisher': None, 'rating': None, 'read_metadata_from_opf': '/var/folders/pq/6qqzm60d6pg555vgfqns87400000gn/C/calibre_5.7.2_tmp_ops5_zhr/5zmn73yr.opf', 'remove_fake_margins': True, 'remove_first_image': False, 'remove_paragraph_spacing': False, 'remove_paragraph_spacing_indent_size': 1.5, 'renumber_headings': True, 'replace_scene_breaks': '', 'search_replace': '[]', 'series': None, 'series_index': None, 'smarten_punctuation': False, 'sr1_replace': None, 'sr1_search': None, 'sr2_replace': None, 'sr2_search': None, 'sr3_replace': None, 'sr3_search': None, 'start_reading_at': None, 'subset_embedded_fonts': False, 'tags': None, 'timestamp': None, 'title': None, 'title_sort': None, 'toc_filter': None, 'toc_threshold': 6, 'toc_title': None, 'transform_css_rules': '[]', 'unsmarten_punctuation': True, 'unwrap_lines': True, 'use_auto_toc': False, 'verbose': 2} Running file type plugin DeDRM failed with traceback: Traceback (most recent call last): File "calibre_plugins.dedrm.__init__", line 534, in KindleMobiDecrypt book = k4mobidedrm.GetDecryptedBook(path_to_ebook,kindleDatabases,androidFiles,serials,pids,self.starttime) File "calibre_plugins.dedrm.k4mobidedrm", line 238, in GetDecryptedBook mb.processBook(totalpids) File "calibre_plugins.dedrm.mobidedrm", line 476, in processBook raise DrmException("No key found in {0:d} keys tried.".format(len(goodpids))) calibre_plugins.dedrm.mobidedrm.DrmException: No key found in 4 keys tried. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "calibre/customize/ui.py", line 172, in _run_filetype_plugins File "calibre_plugins.dedrm.__init__", line 627, in run decrypted_ebook = self.KindleMobiDecrypt(path_to_ebook) File "calibre_plugins.dedrm.__init__", line 577, in KindleMobiDecrypt raise DeDRMError("{0} v{1}: Ultimately failed to decrypt after {2:.1f} seconds. Read the FAQs at Harper's repository: https://github.com/apprenticeharper/DeDRM_tools/blob/master/FAQs.md".format(PLUGIN_NAME, PLUGIN_VERSION,time.time()-self.starttime)) calibre_plugins.dedrm.DeDRMError: DeDRM v7.0.0: Ultimately failed to decrypt after 0.6 seconds. Read the FAQs at Harper's repository: https://github.com/apprenticeharper/DeDRM_tools/blob/master/FAQs.md DeDRM v7.0.0: Trying to decrypt o6q8nezt.azw MobiDeDrm v1.0. Copyright © 2008-2020 The Dark Reverser, Apprentice Harper et al. Decrypting Kindle Format 8 ebook: Proto Zoa Got SerialNumber from database default_key Got UsernameHash from database default_key Found 4 keys to try after 0.0 seconds Crypto Type is: 2 DeDRM v7.0.0: Failed to decrypt with error: No key found in 4 keys tried. DeDRM v7.0.0: Looking for new default Kindle Key after 0.0 seconds Found k4Mac kinf2018 file: /Users/pengy/Library/Application Support/Kindle/storage/.kinf2018 trying username b'pengy' on file /Users/pengy/Library/Application Support/Kindle/storage/.kinf2018 trying IDString: b'b1d87f206bbd' Decrypted key file using IDString 'b1d87f206bbd' and UserName 'pengy' DeDRM v7.0.0: Found 1 new key MobiDeDrm v1.0. Copyright © 2008-2020 The Dark Reverser, Apprentice Harper et al. Decrypting Kindle Format 8 ebook: Proto Zoa Got SerialNumber from database default_key_1 Got UsernameHash from database default_key_1 Found 4 keys to try after 0.5 seconds Crypto Type is: 2 DeDRM v7.0.0: Ultimately failed to decrypt after 0.5 seconds. Read the FAQs at Harper's repository: https://github.com/apprenticeharper/DeDRM_tools/blob/master/FAQs.md Running file type plugin DeDRM failed with traceback: Traceback (most recent call last): File "calibre_plugins.dedrm.__init__", line 534, in KindleMobiDecrypt book = k4mobidedrm.GetDecryptedBook(path_to_ebook,kindleDatabases,androidFiles,serials,pids,self.starttime) File "calibre_plugins.dedrm.k4mobidedrm", line 238, in GetDecryptedBook mb.processBook(totalpids) File "calibre_plugins.dedrm.mobidedrm", line 476, in processBook raise DrmException("No key found in {0:d} keys tried.".format(len(goodpids))) calibre_plugins.dedrm.mobidedrm.DrmException: No key found in 4 keys tried. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "calibre/customize/ui.py", line 172, in _run_filetype_plugins File "calibre_plugins.dedrm.__init__", line 627, in run decrypted_ebook = self.KindleMobiDecrypt(path_to_ebook) File "calibre_plugins.dedrm.__init__", line 577, in KindleMobiDecrypt raise DeDRMError("{0} v{1}: Ultimately failed to decrypt after {2:.1f} seconds. Read the FAQs at Harper's repository: https://github.com/apprenticeharper/DeDRM_tools/blob/master/FAQs.md".format(PLUGIN_NAME, PLUGIN_VERSION,time.time()-self.starttime)) calibre_plugins.dedrm.DeDRMError: DeDRM v7.0.0: Ultimately failed to decrypt after 0.5 seconds. Read the FAQs at Harper's repository: https://github.com/apprenticeharper/DeDRM_tools/blob/master/FAQs.md InputFormatPlugin: MOBI Input running on /var/folders/pq/6qqzm60d6pg555vgfqns87400000gn/C/calibre_5.7.2_tmp_ops5_zhr/o6q8nezt.azw Found KF8 MOBI of type 'standalone' Traceback (most recent call last): File "runpy.py", line 194, in _run_module_as_main File "runpy.py", line 87, in _run_code File "site.py", line 39, in File "site.py", line 35, in main File "calibre/utils/ipc/worker.py", line 216, in main File "calibre/gui2/convert/gui_conversion.py", line 41, in gui_convert_override File "calibre/gui2/convert/gui_conversion.py", line 28, in gui_convert File "calibre/ebooks/conversion/plumber.py", line 1108, in run File "calibre/customize/conversion.py", line 245, in __call__ File "calibre/ebooks/conversion/plugins/mobi_input.py", line 47, in convert File "calibre/ebooks/mobi/reader/mobi8.py", line 86, in __call__ File "calibre/ebooks/mobi/reader/mobi6.py", line 158, in check_for_drm calibre.ebooks.DRMError: Proto Zoa ```

ElleKayEm commented 3 years ago

You deleted and re-downloaded the books and removed and re-added them to calibre?

I don't think the version of Kindle for Mac makes any difference as long as you have KFX disabled. I'd give DeDRM 6.8.1/calibre 4.23 a try.

harshita-gupta commented 3 years ago

I am in the same boat. Freshly installed and registered Kindle version (1.30) from the Mac App Store. Mac OS X 11.1. Successfully disabled KFX using sudo chmod. located the .azw file in my Kindle folder. I've installed the latest pre-release of DeDRM (7.0.0) and Obok DeDRM (6.7.0) [my understanding is that Obok is not necessary but thought I'd include it just in case]. Re-started Calibre. Deleted previously failed attempt at importing book. This is all on the same computer, same user account.

Importing the .azw file still does nothing to remove the DRM. When I try to convert the book to epub Calibre complains and says that it is DRM-protected. Sad :(

I cannot figure out where to find the DeDRM logs. If someone can let me know where to find them I'm happy to post the log.

harshita-gupta commented 3 years ago

Found the log. Welcome any input!

calibre Debug log
calibre 5.7.2  embedded-python: True is64bit: True
macOS-10.16-x86_64-i386-64bit Darwin ('64bit', '')
('Darwin', '20.2.0', 'Darwin Kernel Version 20.2.0: Wed Dec  2 20:39:59 PST 2020; root:xnu-7195.60.75~1/RELEASE_X86_64')
Python 3.8.5
OSX: ('10.16', ('', '', ''), 'x86_64')
Interface language: None
Successfully initialized third party plugins: DeDRM (7, 0, 0) && Obok DeDRM (6, 7, 0)
calibre 5.7.2  embedded-python: True is64bit: True
macOS-10.16-x86_64-i386-64bit Darwin ('64bit', '')
('Darwin', '20.2.0', 'Darwin Kernel Version 20.2.0: Wed Dec  2 20:39:59 PST 2020; root:xnu-7195.60.75~1/RELEASE_X86_64')
Python 3.8.5
OSX: ('10.16', ('', '', ''), 'x86_64')
Interface language: None
Successfully initialized third party plugins: DeDRM (7, 0, 0) && Obok DeDRM (6, 7, 0)
devicePixelRatio: 2.0
logicalDpi: 72.0 x 72.0
physicalDpi: 92.0000013817014 x 92.00000138170138
Using calibre Qt style: True
[0.00] Starting up...
[0.03] Showing splash screen...
[0.27] splash screen shown
[0.27] Initializing db...
[0.28] db initialized
[0.28] Constructing main UI...
[0.93] main UI initialized...
[0.93] Hiding splash screen
Starting QuickView
DeDRM v7.0.0: Trying to decrypt B084M1NCF7_EBOK.azw
Using Library AlfCrypto Python
Using Library AlfCrypto Python
MobiDeDrm v1.0.
Copyright © 2008-2020 The Dark Reverser, Apprentice Harper et al.
Decrypting Kindle Format 8 ebook: What Tech Calls Thinking (FSG Originals x Logic)
Found 0 keys to try after 0.1 seconds
Crypto Type is: 2
DeDRM v7.0.0: Failed to decrypt with error: No key found in 0 keys tried.
DeDRM v7.0.0: Looking for new default Kindle Key after 0.1 seconds
Found k4Mac kinf2018 file: /Users/Harshita/Library/Containers/com.amazon.Kindle/Data/Library/Application Support/Kindle/storage/.kinf2018
trying username  b'Harshita'  on file  /Users/Harshita/Library/Containers/com.amazon.Kindle/Data/Library/Application Support/Kindle/storage/.kinf2018
trying IDString: b'5d5a674fe638'
Decrypted key file using IDString '5d5a674fe638' and UserName 'Harshita'
DeDRM v7.0.0: Found 1 new key
MobiDeDrm v1.0.
Copyright © 2008-2020 The Dark Reverser, Apprentice Harper et al.
Decrypting Kindle Format 8 ebook: What Tech Calls Thinking (FSG Originals x Logic)
Got SerialNumber from database default_key_1
Got UsernameHash from database default_key_1
Found 4 keys to try after 0.6 seconds
Crypto Type is: 2
DeDRM v7.0.0: Ultimately failed to decrypt after 0.6 seconds. Read the FAQs at Harper's repository: https://github.com/apprenticeharper/DeDRM_tools/blob/master/FAQs.md
Running file type plugin DeDRM failed with traceback:
Traceback (most recent call last):
  File "calibre_plugins.dedrm.__init__", line 534, in KindleMobiDecrypt
    book = k4mobidedrm.GetDecryptedBook(path_to_ebook,kindleDatabases,androidFiles,serials,pids,self.starttime)
  File "calibre_plugins.dedrm.k4mobidedrm", line 238, in GetDecryptedBook
    mb.processBook(totalpids)
  File "calibre_plugins.dedrm.mobidedrm", line 476, in processBook
    raise DrmException("No key found in {0:d} keys tried.".format(len(goodpids)))
calibre_plugins.dedrm.mobidedrm.DrmException: No key found in 0 keys tried.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "calibre/customize/ui.py", line 172, in _run_filetype_plugins
  File "calibre_plugins.dedrm.__init__", line 627, in run
    decrypted_ebook = self.KindleMobiDecrypt(path_to_ebook)
  File "calibre_plugins.dedrm.__init__", line 577, in KindleMobiDecrypt
    raise DeDRMError("{0} v{1}: Ultimately failed to decrypt after {2:.1f} seconds. Read the FAQs at Harper's repository: https://github.com/apprenticeharper/DeDRM_tools/blob/master/FAQs.md".format(PLUGIN_NAME, PLUGIN_VERSION,time.time()-self.starttime))
calibre_plugins.dedrm.DeDRMError: DeDRM v7.0.0: Ultimately failed to decrypt after 0.6 seconds. Read the FAQs at Harper's repository: https://github.com/apprenticeharper/DeDRM_tools/blob/master/FAQs.md
[14.93] splash screen hidden
[14.93] Started up in 14.93 seconds with 0 books
Added What Tech Calls Thinking (FSG Originals x Logic) to db in: 0.0
Added 1 books in 1.1 seconds
Traceback (most recent call last):
  File "calibre/gui2/ui.py", line 134, in __init__
  File "calibre/gui2/ui.py", line 152, in init_iaction
  File "calibre/customize/__init__.py", line 614, in load_actual_plugin
  File "importlib/__init__.py", line 127, in import_module
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "calibre/customize/zipplugin.py", line 192, in exec_module
  File "calibre_plugins.obok_dedrm.action", line 24, in <module>
    from calibre_plugins.obok_dedrm.dialogs import (SelectionDialog, DecryptAddProgressDialog,
  File "calibre/customize/zipplugin.py", line 192, in exec_module
  File "calibre_plugins.obok_dedrm.dialogs", line 30, in <module>
    from calibre_plugins.obok_dedrm.utilities import (SizePersistedDialog, ImageTitleLayout,
  File "calibre/customize/zipplugin.py", line 192, in exec_module
  File "calibre_plugins.obok_dedrm.utilities", line 10, in <module>
    from StringIO import StringIO
ModuleNotFoundError: No module named 'StringIO'

lauraannwilliams commented 3 years ago

The obok one works very well - I use it on my Kobo and Smashwords books which download through adobe digital images. You don't need it if you're only using Amazon

ZolaLa9 commented 3 years ago

@harshita-gupta — try Calibre 4.23 and tools v6.8.1

If that doesn't work, try downloading K4Mac 1.30.0 from Amazon's website (click on Download for PC & Mac) rather from the Mac App Store. Be sure to delete /Users/Harshita/Library/Containers/com.amazon.Kindle folder.

ElleKayEm commented 3 years ago

Obok is only for books from the Kobo Desktop app or a Kobo ereader.

harshita-gupta commented 3 years ago

@ZolaLa9 tried with the older Calibre and tools versions, and it worked – thanks for the suggestion.

lauraannwilliams commented 3 years ago

Works for me as well with the older versions. thanks

ElleKayEm commented 3 years ago

What I've been recommending all along. :)

apprenticeharper / DeDRM_tools