search

apprenticeharper / DeDRM_tools

DeDRM tools for ebooks
14.56k stars 1.52k forks source link

Support kindle kfx format #38

Closed extesy closed 4 years ago

extesy commented 9 years ago

Is there any chance to support decrypting new kfx format? http://the-digital-reader.com/2015/08/11/kindle-ebooks-with-improved-typography-use-new-kfx-file-format/

j-howell commented 6 years ago

@voltagex has combined the pull request into a plugin zip file that can be used for testing. (See the comments of #458.)

The KFX Input plugin for calibre also needs to be installed to allow creation of a KFX-ZIP archive when a single KFX file is selected for import into calibre and to allow conversion to more useful formats.

I gave it a try on an assortment of KFX books from both Kindle for PC 1.21 and a Kindle running 5.9.4 firmware. I had to select books using the "Add books from a single directory" option in calibre, since calibre would not allow import of books KFX format from the Device view.

All but a few were able to have their DRM removed successfully! I will try those that didn't work in debug mode and see if I can find what is going wrong with them. (I don't have any more time to spend on this at the moment.)

While testing I found a problem in the KFX Input plugin with the handling of e-book samples. I will work on a fix for that.

cemeyer commented 6 years ago

@j-howell For the few that were unsuccessful, perhaps their license_type != "Purchase" (DeDRM_plugin/kfxdedrm.py)? That could be expanded (or dropped). (I don't know what possible values that string can have.)

j-howell commented 6 years ago

For the few that were unsuccessful, perhaps their license_type != "Purchase"

You were right, @cemeyer. It failed for books from Kindle Unlimited:

DeDRM v6.5.5: Failed to decrypt with error: This book is licensed as KindleUnlimited. These tools are intended for use on purchased books.

And for library books borrowed through OverDrive:

DeDRM v6.5.5: Failed to decrypt with error: This book is licensed as PublicLibraryLending. These tools are intended for use on purchased books.

I suppose @apprenticeharper will have to decide whether or not to keep that limitation when integrating the changes.

It also doesn't work for free Indic dictionaries in KFX format. Those use a different variant of KFX:

The .kfx-zip archive does not contain an encrypted DRMION file

It did work properly for all of the purchased books I tried.

toXel commented 6 years ago

Isn't this protection intended behavior? Why should it be changed for KFX files?

mafrosis commented 6 years ago

I've been testing the KFX branch written by @tomthumb1997. I see the following error with all the books sync'd down through the latest Kindle for Mac:

The .kfx DRMION file cannot be decrypted by itself. A .kfx-zip archive containing a DRM voucher is required.

The code which produces that message checks the first 8 bytes of file for the special chars "DRMION". I checked all my azw files and they all have this same first 8 bytes!

head -c 8 <azw_file>

Does anyone know more about how these tools work?

j-howell commented 6 years ago

Isn't this protection intended behavior? Why should it be changed for KFX files?

The explicit license type used in KFX format means that the check for purchased books will be more reliably enforced than for other Kindle formats. Personally, I think that is a good thing. Others may not agree.

cemeyer commented 6 years ago

@mafrosis I think vouchers can be stored separately from kfx files by the Kindle software, but @ tomthumb1997 's plugin only handles kfx-zip with embedded voucher files. See https://gist.github.com/apprenticenaomi/b0103db2ac134cb51a3ead92f1d99f3a#file-main-py-L30 for example (it seems to use a weird callback scheme to locate appropriately named ".voucher" files in the same directory as the book files). Don't know when or how Kindle software decides to store zips or separate files.

j-howell commented 6 years ago

Does anyone know more about how these tools work?

The KFX support for these DRM Tools implemented by @tomthumb1997 requires a kfx-zip archive as its input, not a DRMION file.

A book in KFX format is composed of multiple files, including a DRM voucher with encryption keys. In order to remove DRM all of the files that make up the book are required, not just the encrypted DRMION file. In the Handling KFX files in Calibre thread on MobileRead it was decided that the best way to deal with this is to combine all of the required files into an archive and give it a .kfx-zip extension.

When I first created the KFX Input plugin for calibre it required the user to zip the component files of each KFX book to produce a kfx-zip archive. I later made a change to have that plugin gather all of the required files and produce the kfx-zip automatically to make it easier to use.

So if you install the KFX Input plugin for calibre then it should just work as expected. When you select the main KFX file for import, the plugin will combine it with the other required files into a kfx-zip, which will then be passed to the DeDRM plugin. After that completes the KFX Input plugin will take the de-DRMed result and combine it into a monolithic KFX file that will be added to your calibre library.

The resulting unencrypted KFX file can then be sideloaded to various Kindle apps and devices or it can be converted to other formats within calibre, again using the KFX Input plugin.

mafrosis commented 6 years ago

@cemeyer Thanks for the info. Very useful

@j-howell I don't use Calibre, but again very useful info.

cemeyer commented 6 years ago

@j-howell Oh, KFX-zip is entirely something of your creation, not an Amazon thing? I definitely think we should support vanilla kfx files from Amazon. We can do both. :-)

The explicit license type used in KFX format means that the check for purchased books will be more reliably enforced than for other Kindle formats. Personally, I think that is a good thing. Others may not agree.

Seems harmless, but you know crapware like EPUBOR is going to remove the check.

j-howell commented 6 years ago

Oh, KFX-zip is entirely something of your creation, not an Amazon thing?

It was decided that since books in KFX format are spilt into multiple files and calibre handles all books as single files that combining the KFX files that make up a book into a zip archive was the best way of making it work within that restriction of calibre.

The DRMION file contains only part of the content of the book. The KFX Input plugin will merge all of the component files in the KFX-ZIP into a single KFX file, but that can't be done until after DRM has been removed from the DRMION file.

apprenticeharper commented 6 years ago

Thanks, everyone for all your work on this. I've merged the pull request, tweaked some readmes and version numbers, and v6.6.0 has now been released. It needs testing and no doubt some bug fixes. And the FAQs could do with an update. But that's all I have time to do today. Anything else will have to wait until I have some more free time.

wallyreport commented 4 years ago

I am using Devaun, and having the same problem.

Linux myserver 4.19.0-10-amd64 #1 SMP Debian 4.19.132-1 (2020-07-24) x86_64 GNU/Linux

I cannot seem to upgrade calibre beyond 3.9.1.

I have python 2.7 installed.

Roze20 commented 4 years ago

Contact Calibre official for help please.

On Sat, Oct 31, 2020 at 3:16 AM wallyreport notifications@github.com wrote:

I am using Devaun, and having the same problem.

Linux myserver 4.19.0-10-amd64 #1 https://github.com/apprenticeharper/DeDRM_tools/issues/1 SMP Debian 4.19.132-1 (2020-07-24) x86_64 GNU/Linux

I cannot seem to upgrade calibre beyond 3.9.1.

I have python 2.7 installed.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/apprenticeharper/DeDRM_tools/issues/38#issuecomment-719746779, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADU4EK2B343TC7MJHQGLU2DSNMGHVANCNFSM4BPYMY2Q .

ElleKayEm commented 4 years ago

@wallyreport I'd ask about calibre issues over in the calibre forum at MobileRead.