Support kindle kfx format

[extesy]

Is there any chance to support decrypting new kfx format? http://the-digital-reader.com/2015/08/11/kindle-ebooks-with-improved-typography-use-new-kfx-file-format/

[apprenticeharper]

It would be great to support the new format. And the new azw3/azw6 combo files too. But reverse engineering a new DRM scheme and format is a hard task, especially one that's only on physical kindles.

If anyone comes up with a method, I'll add it to the tools as soon as possible.

[extesy]

According to http://www.mobileread.com/forums/showthread.php?t=263902 it appears that DRM scheme in kfx format is not device-dependent. It's possible to simply move the files between different Kindle devices (even without associated Amazon account) and it still works. So basically it's just a reversible encryption with all required for decryption data stored in those files.

[mbrown3]

Would LOVE for this to happen.

[Roze20]

Decrypting new kfx format is possible, of course.

But firstly you need to sync your books via Kindle for PC/Mac, this sync will make these books downloaded as AZW format, then remove drm from AZW please.

As what I known, there is no good solution to decrypte azw6 format books. Books what we get will be KFX (new).

https://garydadic.medium.com/convert-kindle-kfx-to-epub-pdf-free-on-mac-10-15-catalina-included-7ea1d7d2b609

[ElleKayEm]

angelana,

For AZW6, if you have an e-ink kindle, then you can download an AZW3 for it from the website and then decrypt that.

[Roze20]

@ElleKayEm

Thank you so much, this solution helped me.

[gitasong]

Amazon is not allowing me to download the AZW3 for a book I've purchased in AZW6 format. It keeps giving me the following error message:

"You have selected content that is not compatible with any of your registered devices. Please remove incompatible content from selection for eligible devices to appear."

My registered devices include Kindle for Mac, an old Kindle Paperwhite (1st generation, I think), and Kindle for Android.

[jessehouwing]

Azw6 is a special format for graphic novels, comics and magazines and is only available on specific devices. Since the graphic content is stored 'outside' of the book file, it can't be saved as an awz3 file. The file would simply become too large.

[gitasong]

That's not the case for this book. This book simply uses a nice font with the AZW6 "advanced typesetting" feature. There's zero reason why it couldn't be offered in AZW3. This is a case of Amazon wanting to control distribution and/or make an extra buck by forcing you to upgrade their products.

[ElleKayEm]

I'm happy to discover that all my comics (AZW6 when downloaded directly to my Paperwhite) will now download to Kindle for PC as AZW3. No need to "download & transfer via USB" for my Paperwhite anymore. On the product pages they say they're also available for Kindle for Mac. Is that not the case for your book, gitasong?

[gitasong]

It may be, but my machine is too old; Kindle for Mac is no longer supported (and the old versions no longer work, believe it or not).

[ElleKayEm]

The 1st generation Paperwhite is also listed. (They call it 5th gen.) Even Kindle Keyboard. Are you sure that Paperwhite is still registered to that account?

[gitasong]

Maybe it's 2nd generation; I'm not sure. And yes—it's still registered. I was on the phone with Amazon Kindle support for like 45 minutes today. The issue seems to be that the publisher hasn't issued this book in AZW3 format yet. They're giving me a refund. Anyway, it would really nice to figure out how to circumvent this issue.

[ElleKayEm]

That sounds pretty flimsy to me! Blame the publisher! But at least they're good about refunds. I usually get comics from Google. They're often available in both epub and pdf (one purchase) and can be DeDRMed.

[apprenticeharper]

Now that KFX is being downloaded by Kindle for PC 1.19, I hope that some developers will work on reverse engineering it. I'd love to hear from anyone doing this.

[j-howell]

In the mean time I have a suggestion. The latest version of K4PC uses the same extension, .azw, for all book formats. Encrypted KFX containers begin with an 8 byte signature that includes the string "DRMION". An explicit error message for this case would avoid some confusion in diagnosing DRM removal problems.

Edit: The exact string is: b'\xeaDRMION\xee'

[apprenticeharper]

That's a good idea. It might be worth doing another version of the tools just to cut down on questions at the blog. And there is an Obok update that should be released in a full version. Hmm...

[sporks5000]

Is there anyone that donations could be made to in order to increase the likelihood of this occuring?

[apprenticeharper]

I don't accept donations for these tools. I hope that someone is working on KFX decryption/decoding, but I don't know of anyone at the moment.

[tmthywynn8]

I notice that these KFx formats include a separate file with a .md extension. Is there a possibility that they are just encrypted markdown files? If so, is there a possibility of just decrypting the .md files? I understand this is a separate issue from the KFX one, but it's worth a shot to ask.

[j-howell]

In this case "md" is short for metadata. It holds information about the book, such as author, title and cover image. This file is not encrypted, but it is in KFX binary container format, making it difficult to interpret.

[apprenticeharper]

Note that Epubor Ultmate does not remove DRM from KFX format files. This is unsurprising, since it's built on the scripts released here.

[ElleKayEm]

Don't waste your money on Epubor. It's simply a repackaging of the tools freely provided here.

[j-howell]

For those interested in the workings of KFX, Amazon has documentation for Ion, the data format that KFX is based on, online at https://amznlabs.github.io/ion-docs/

Reference implementations of Ion in Java, C and Python can be found at https://github.com/amznlabs

[apprenticeharper]

"lulzkabulz" has reverse-engineered the KFX encryption from Kindle for PC and produced sample decryption code in Pascal. Their code removes the decryption only, leaving a document in the Ion format (see j-howell's comment above.)

I have added a copy of lulzkabulz's code to the repository.

The conversion of the decryption code to python is essentially mechanical, and conversion from Ion to other formats will probably be relatively straight-forward.

I'll be having a go at the Pascal->Python conversion myself, but I don't have much free time to spend on it. Contributions to this, or the Ion -> ePub conversion task are very welcome.

[apprenticenaomi]

Hi,

I had a go at the mechanical Pascal to Python conversion myself. I've tested it with a KFX ebook and can successfully decrypt it (see main.py decrypt()). The various other print / debugging methods seem to work ok too.

Conversion from the output to EPUB may be more difficult, but it looks like Calibre already has some support there.

Thanks lulzkabulz for the original work.

[jcsalomon]

@apprenticenaomi, what does a decrypted e-book in Ion format look like?

[apprenticenaomi]

@jcsalomon It's a CONT (container) file with a bunch of ENTYs (entities). The book contents are embedded in there as uncompressed UTF-8. I haven't looked at the structure too much.

You can "buy" a free book from Amazon, download it, and try it out yourself. Nothing in the file looks personally identifying, but I'm not about to upload a DRM-stripped book :-).

[apprenticeharper]

Many thanks for the python conversion! If anyone fancies having a go at convertingwhat I'm assuming is an ION format book (see the comment by j-howell), I suggest that conversion to epub (version 2) might make most sense and be easiest.

[jcsalomon]

Everyone involved in this thread probably already has seen something like this, but for the benefit of folks reading this later: @j-howell has uploaded a snippet of a KFX file (from Kindle Previewer so no DRM involved) unpacked to Ion at https://mobileread.com/forums/showthread.php?t=288540#post3552191.

[j-howell]

Terminology: A KFX book is split over multiple files each using KFX container format. A KFX container is conceptually similar to the PDB format used as the basis of MOBI and KF8. It holds a list of data records known as fragments or entities. The entities are data structures encoded in Amazon's Ion binary format. Amazon has an equivalent Ion text format, a superset of JSON, which can encode the same data in a human-readable fashion. Synonyms (more or less) for KFX (kindle format 10) are YJ (yellow jersey), KRF (?), and AZW8.

The process of converting a book from EPUB to KFX is handled by a very large Java program (KFXGenApp). It works much like a compiler, transforming the original EPUB into rendering instructions. In the process any information in the original EPUB that is not relevant to how the book is rendered into pages is discarded. The result bears little resemblance to the HTML/CSS within an EPUB. This is far different from KF8, which is just the original EPUB content packed into a MOBI container.

I believe that it will take a large effort to develop software that can convert KFX back into EPUB. (And even if successful the result will be inferior to that which can be obtained by just unpacking a KF8.) I propose that this effort is better left to an outside project, such as calibre, which is not hampered by the potential legal issues that come from being part of DRM removal software.

[apprenticenaomi]

The converted Pascal program can parse and unpack the KFX container format, but it doesn't do much beyond that. Amazon's Python reference implementation might be the best way forward.

It does seem most reasonable for Calibre to do conversion, and leave DRM to this project.

[j-howell]

I have looked at the Pascal program that was posted and it has multiple functions that do different things. This may be causing confusion.

One function implements the extraction of encryption keys from Kindle for PC. It duplicates existing functionality of the DRM tools.

Another function takes an encrypted KFX file (DRMION) and the extracted keys and removes the encryption, resulting in an unencrypted KFX container. That leaves a working KFX book without DRM. That function would be nice to have in the DRM removal toolkit.

A separate function decodes the KFX container format and the data structures it contains in Ion binary format and outputs a human readable version of the data. That is useful for delving into the workings of KFX and could provide a starting point for conversion of KFX to EPUB.

[lizzard]

aph3rson, do you have a link to the Kindle 1.17 Mac version download page? Thanks.

[ElleKayEm]

Kindle for Mac 1.17: https://s3.amazonaws.com/kindleformac/44182/KindleForMac-44182.dmg

(Mentioned here in the FAQs.)

[j-howell]

Someone asked on Apprentice Alf’s Blog about removing DRM from KFX books that were obtained via Kindle Unlimited and the response was that the DRM removal Tools are only intended for use on purchased books.

KFX voucher files have a license_type field that indicates whether that particular book is a rental, loan or purchase. I suggest that this field be checked if/when KFX DRM removal becomes part of the Tools.

[apprenticeharper]

A very good suggestion. Thanks.

[j-howell]

Two bits of KFX news:

1. I have released a calibre plugin that can convert from KFX to other formats. At present it is difficult to use due to the lack of KFX DRM removal support in a calibre plugin.

2. EPUBOR has announced that they now support DRM removal of KFX format. I assume that they incorporated the code that was posted here, but never integrated in the DRM TOOLS. It is probably only a matter of time before they add a KFX conversion function too. :(

[apprenticeharper]

Thank you again for all you work on this. My development time is limited at the moment, but I would welcome pull requests incorporating the new KFX decryption capability into the tools.

[j-howell]

The download links for Kindle for PC/Mac 1.17 from Amazon listed under "A Recent Change to Kindle for PC/Kindle for Mac" in the FAQ are no longer operational.

[jostyee]

@j-howell dmg backup: https://d.pr/f/94aJrl

[j-howell]

I don't need the file myself, but others may find it useful.

I did a write-up that describes ways to work around KFX. It includes checksums to verify that you have found the correct older Kindle for PC or Mac file. (The one above looks good, for now.)

https://www.mobileread.com/forums/showthread.php?t=283371

[vdcbb]

Is this the place to watch for when KFX decryption capability becomes available?

[ElleKayEm]

This GitHub is the place to watch for when it gets integrated into the tools (and any other updates).

[bombcheck]

While waiting for this to happen, using the old K4PC-App to get an AZW-Version of the book should do the trick...

[garfieldthecat]

EPUBOR has announced that they now support DRM removal of KFX format.

That's right. I have tried it: conversion and decryption work, but the final output is a text file that loses all the formatting, chapters, etc. I have contacted them to understand if there is a way around it.

[apprenticeharper]

Apologies that I have had no time to work on this yet. Pull requests welcome.

[j-howell]

I see a pull request for KFX support. I have been waiting for someone to integrate that capability into these tools. Hopefully it works. I will give it a try.

[ElleKayEm]

@chorichori

Start your own issue. You shouldn't have any trouble using DeDRM with an AZW3 file.

[cemeyer]

Re j-howell, the alluded to PR is #458 . Let us know how it goes!