noDRM / DeDRM_tools

DeDRM tools for ebooks
7.37k stars 324 forks source link

Any timeline to DeDrm to work with latest Kindle for PC Version? #240

Open MadMik1964il opened 1 year ago

MadMik1964il commented 1 year ago

Question / bug report

As for today, Amazon doesn't let older versions of Kindle for PC to download files. And current version of DeDrm doesn't work with new ver 1.39 files (files are imported as KFX-ZIP). Please help!!!

Which version of Calibre are you running?

6.1

Which version of the DeDRM plugin are you running?

v10.0.3

If applicable, which version of the Kindle software are you running?

1.39

Log output

No response

noDRM commented 1 year ago

Does this apply to all books? I believe there's some books that require the newest Kindle 4 PC, but most books should still work with the old one.

federicorosso1993 commented 1 year ago

Does this apply to all books? I believe there's some books that require the newest Kindle 4 PC, but most books should still work with the old one.

I tried a few, right now I'm able to download books with the old version but not manga/comics. Luckily the manga I bought even in the new version come without KFX (I was able to do it both for italian manga and Claymore that I own the english version). Still last week I was able to download manga with the old version so I think they will change it in a short amount of time.

iamsivil commented 1 year ago

Seems like this may be a relatively new thing. I was able to download a book that released approximately 30 hours ago on Kindle for PC 1.17 as of this morning but 2 books released since then (one 20 hours ago and the other 6 hours ago) were blocked from being downloaded and required upgrading to the latest Kindle version.

j-howell commented 1 year ago

I am curious what Amazon has changed. I tried some very recently published books and they all downloaded for me. Can someone with this problem provide links to some of the books that are not working for you using K4PC 1.17?

MadMik1964il commented 1 year ago

Actually I was using 1.26 for a year or so and it was fine. Until yesterday. Tried to get few books, and it is not working anymore. I returned them, but we do need a solution.

ElleKayEm commented 1 year ago

Still hanging in here with Kindle for PC 1.17. See noDRM's response to a similar question here: https://github.com/noDRM/DeDRM_tools/issues/239#issuecomment-1371083530

MadMik1964il commented 1 year ago

Well. looks I might need to buy Kindle device, damn it...

ElleKayEm commented 1 year ago

That won't help either if the book is only available in the latest KFX format.

UnKnoWn-Consortium commented 1 year ago

I am not entirely sure but it appears they have switched off azw download at old Kindle PC versions for books published after 2nd January 2023 (give or take).

Edit: AZW download for transfer over USB to Kindle devices are still working as of now. In other words we lose access to high-res resource packs (when available)...

MadMik1964il commented 1 year ago

Well, looks I have to find spare $200 bucks to buy the device (used my phone for reading so far)...

ElleKayEm commented 1 year ago

Again, you don't have that option if the book is only available in KFX format. And if you're going to get a Kindle just for download and transfer via usb, you can get one for a lot less.

j-howell commented 1 year ago

I am curious what Amazon has changed. I tried some very recently published books and they all downloaded for me.

I take back what I wrote earlier. Version 1.17 is no longer letting me download newly published books.

ElleKayEm commented 1 year ago

You can still download older books with 1.17? Can you download newly published books via one of the other methods for avoiding kfx? Or at least the latest kfx?

j-howell commented 1 year ago

You can still download older books with 1.17? Can you download newly published books via one of the other methods for avoiding kfx? Or at least the latest kfx?

Only newly published books seem to be affected. New purchases of books published last year still work in 1.17. Download & transfer targeted to an e-ink Kindle still works even for the newly published ones. I will try other app versions tomorrow.

j-howell commented 1 year ago

I tested Kindle for PC versions from 1.17 through the current 1.39. Only version 1.39 is capable of downloading newly published books.

ElleKayEm commented 1 year ago

Thanks for testing and reporting back.

VorpalQ commented 1 year ago

Again, you don't have that option if the book is only available in KFX format. And if you're going to get a Kindle just for download and transfer via usb, you can get one for a lot less.

I decided to test this, since I have an old Kindle Paperwhite laying around. So I dusted it off and registered it to my Amazon account. I bought and downloaded a book that was published on Jan 4th, and so could only be downloaded on 1.39. When I used the Kindle PC App, the book came in KFX-ZIP and could not have the DRM removed. When I instead used the 'Download & Transfer via USB' option from 'Content & Device Management' on Amazon's website, it downloaded an AZW3 file that I was then able to import to Calibre and remove the DRM from.

MadMik1964il commented 1 year ago

Thanks for testing. I"ll go look for old Kindle on ebay.

ElleKayEm commented 1 year ago

Yes, it is only blocked from "download & transfer" if it's a book which is only available in the latest KFX format. A very small category usually expensive textbooks and certain languages.

amnotbananaama commented 1 year ago

Echoing the OP, now that the problem is more clearly defined. While the current issue (some Kindle books published after 1/3/2022 are KFX format only and cannot be stripped by DeDRM) is minor at present, the number of inaccessible books will continue to grow with time. Not to mention, there remains the possibility of Amazon switching to KFX format only for all books in the future. Is there any foreseeable way that DeDRM could support the KFX format in the future?

Of course, I realize that is a lot to ask, and I would like to sincerely thank the maintainer(s) for their work on DeDRM, which has been invaluable to me as a user.

ElleKayEm commented 1 year ago

noDRM said in another issue:

The past has shown that A) figuring out how to break the new Amazon DRMs is a ton of work, and B) Amazon already has multiple new DRM versions "available" that they can switch to in a day or two, so the effort would be totally wasted.

For now, even most new books are still available in the older decryptable format. It's just that the only way to get them in that format is now "download & transfer via usb" for an E-ink Kindle. Only a few books are truly KFX only. On the flip side, there are a few books (manga and comics) which don't come in KFX at all (yet). Those can be downloaded with the latest Kindle for PC and still be DeDRMed. Trouble is all of this could change and it's hard to tell before buying what you might get stuck with.

federicorosso1993 commented 1 year ago

I think once I cannot remove the drm anymore I will switch to buying from kobo since right now Adobe drm is easier to remove but I also think the only 2 ways to deal with this kind of new drm from amazon in the future will be to:

  1. Figure out the pattern to deal with KFX drm (I don't think it's a random/manual way to encode it every time in a really different kind of way... I think there should be a pattern or they would not be able to change it so fast).
  2. Go full OCR so find a way to automate the process to acquire amazon ebooks in image format and OCR them. Right now it's possible to take screenshot from both the Kindle PC app and the kindle website (read.amazon.com) so it should be possible to automate the process. This should be possible for kobo too. The only problem with this is that OCR is not perfect and screenshot quality is not perfect for manga/comics, the good thing is that even without any working tool it will still be possible to do something like that manually.

I don't think it's a good idea to buy old kindle right now, they can change the "download & transfer via usb" anytime they want, just buy from kobo and remove adobe drm instead of amazon drm. And I'm not sure manga and comics will stay KFX-free forever.

j-howell commented 1 year ago
  1. Figure out the pattern to deal with KFX drm (I don't think it's a random/manual way to encode it every time in a really different kind of way... I think there should be a pattern or they would not be able to change it so fast).

Amazon has implemented around 40 different DRM algorithms for KFX. They are built into the various Kindle apps and device firmware as highly obfuscated code.

Each algorithm is identified with the letter V followed by a number. So far DeDRM handles V1 though V10. Amazon is currently using V11, which DeDRM attempts to decrypt but does not do correctly.

As things stand now anyone who does figure out how to defeat the current V11 DRM has a strong incentive to keep it to themself. Keeping it quiet would mean that at least they could make use of it. Making it known to the world would just result in Amazon switching to V12, causing all of that hard work to be for nothing.

In order to defeat KFX DRM in DeDRM all of the thirty or so remaining DRM algorithms would need to be reverse engineered. If any are gotten wrong, which is likely without a way to test them properly, Amazon would immediately use that one for now and then keep adding more new ones over time.

Even if someone managed to get all of the algorithms correct, DRM removal would only work until Amazon deployed software with more new algorithms and blocked downloads to the older software.

noDRM commented 1 year ago

Someone on MobileRead posted that apparently Kindle 4 Android in version 7.6.0.39 can still download books from 2023, and downloads books in one of the older KFX formats where the DRM can still be removed; maybe that info is helpful for some people. Though I have not verified if that does actually work; and you'll need to either run that on an Android emulator or on a rooted Android device to be able to extract the key files. So it's probably not a solution for the average user, and I'd imagine that Amazon would force Android users to update soon, too.

As for the different KFX versions, I have to admit I didn't really look too closely at the Amazon DRM in the past. Interesting that the tools only work up to V11 - looking at the OBFUSCATION_TABLE in ion.py I would have expected that all versions in that list can be DeDRM'ed and that the newer versions of K4PC / KFX would be using version numbers not yet in that list.

But I guess when Amazon wasn't yet using versions >10 there was no way to test that DRM removal so they probably have some additional protections on these versions, making that table useless. Interesting.

j-howell commented 1 year ago

But I guess when Amazon wasn't yet using versions >10 there was no way to test that DRM removal so they probably have some additional protections on these versions, making that table useless. Interesting.

I looked when the last DeDRM stopped working for KFX and the version I saw in use that could not be decoded was V11. Either the table is wrong or there is something more that needs to be done to decode that DRM version.

Just a suggestion, but it would be nice if the code that handles KFX DRM would print out the DRM version number. That would make it possible to tell which version was being used when dealing with reports of failure.

ElleKayEm commented 1 year ago

Even if KFX DRM was figured out, I'd rather have azw3 for conversion to epub. If they make it too difficult to get, I'll say goodbye to Amazon as I did with B&N many years ago. Kobo gets most of my business already.

UPSA007 commented 1 year ago

Ya but the problem with kobo is their prices. Most of the books cost anywhere from 10 to 20 percent higher and sometimes double for no bloody reason. Google Play books also has the same problem. I mean who decides the price publishers or the platforms.

shamanNS commented 1 year ago

@noDRM

Someone on MobileRead posted that apparently Kindle 4 Android in version 7.6.0.39 can still download books from 2023, and downloads books in one of the older KFX formats where the DRM can still be removed; maybe that info is helpful for some people.

Hello from "someone" 😄

ElleKayEm commented 1 year ago

Ya but the problem with kobo is their prices. Most of the books cost anywhere from 10 to 20 percent higher and sometimes double for no bloody reason. Google Play books also has the same problem. I mean who decides the price publishers or the platforms.

I don't see that that often for the books I buy. But Kobo has price matching. You get the difference (plus 10% of the competitor's price) back as credit.

federicorosso1993 commented 1 year ago

I don't see this at all for book and manga that I buy (same price from Amazon and Kobo) but there is no price matching here. I can generate amazon code from my bank account to buy things (and ebooks) on Amazon with a 3% discount... That's why I usually buy from Amazon instead of Kobo (and since kobo is not drm free but I still need to remove the drm I prefer Amazon). But if there will be no way to remove drm from Amazon I will switch to kobo even if I have to pay 3% more is not a big deal.

VorpalQ commented 1 year ago

Again, you don't have that option if the book is only available in KFX format. And if you're going to get a Kindle just for download and transfer via usb, you can get one for a lot less.

I decided to test this, since I have an old Kindle Paperwhite laying around. So I dusted it off and registered it to my Amazon account. I bought and downloaded a book that was published on Jan 4th, and so could only be downloaded on 1.39. When I used the Kindle PC App, the book came in KFX-ZIP and could not have the DRM removed. When I instead used the 'Download & Transfer via USB' option from 'Content & Device Management' on Amazon's website, it downloaded an AZW3 file that I was then able to import to Calibre and remove the DRM from.

An update to my earlier post - I encountered two newly published (and purchased) books today that could not have the DRM removed even when downloaded via the 'Download & Transfer to USB' method as azw3 files. The third new book I got today did not have this problem. I'm not sure what to make of this. Disregard all this, I forgot to enter my S/N for my e-Ink Kindle. :P

ElleKayEm commented 1 year ago

You chose the same E-ink Kindle for all and have that serial number entered in DeDRM? Or was the 3rd one DRM-free already?

VorpalQ commented 1 year ago

You chose the same E-ink Kindle for all and have that serial number entered in DeDRM? Or was the 3rd one DRM-free already?

Nevermind I am an idiot. I hadn't entered my serial number and coincidentally it hadn't been a problem before now :P I'll update my other post accordingly.

deirdrew commented 1 year ago

I am so relieved I kept two really old DXs as registered devices at Amazon (they no longer turn on). I added both serial numbers to the plugin after reading the posts, so now I can just download the problem books to my computer via the more actions>download via USB option at Amazon. Thanks, folks! Image2

MegMM commented 1 year ago

I've confirmed it's only 2023 books that can't be downloaded. 2022 and older books can be; I run KFPC 1.17. But I have noticed a lot of older books being republished.

So I would recommend you download all or as many books as you can ASAP

-Meghan


From: John Howell @.> Sent: Wednesday, January 4, 2023 7:45:01 PM To: noDRM/DeDRM_tools @.> Cc: Subscribed @.***> Subject: Re: [noDRM/DeDRM_tools] Any timeline to DeDrm to work with latest Kindle for PC Version? (Issue #240)

You can still download older books with 1.17? Can you download newly published books via one of the other methods for avoiding kfx? Or at least the latest kfx?

Only newly published books seem to be affected. New purchases of books published last year still work in 1.17. Download & transfer targeted to an e-ink Kindle still works even for the newly published ones. I will try other app versions tomorrow.

— Reply to this email directly, view it on GitHubhttps://github.com/noDRM/DeDRM_tools/issues/240#issuecomment-1371749861, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJLCH7ZE7GZB4ZRWLB3ZLM3WQY7T3ANCNFSM6AAAAAATQU2YLY. You are receiving this because you are subscribed to this thread.Message ID: @.***>

alehel commented 1 year ago

So I downloaded a few comics using the download and transfer method. It allowed me to pick my kindle fire 7 for the download. Problem is, it seems to be a low resolution version. File size is only 16 mb, but through the kindle desktop app I get a 64 mb file. I'm guessing I'm getting a version optimized for a 7 inch tablet. Questions is, if I get a Kindle Fire 10 HD Plus, will I get higher quality comics using the download and transfer method? Anyone have any experience with this?

j-howell commented 1 year ago

So I downloaded a few comics using the download and transfer method. It allowed me to pick my kindle fire 7 for the download. Problem is, it seems to be a low resolution version. File size is only 16 mb, but through the kindle desktop app I get a 64 mb file. I'm guessing I'm getting a version optimized for a 7 inch tablet. Questions is, if I get a Kindle Fire 10 HD Plus, will I get higher quality comics using the download and transfer method? Anyone have any experience with this?

No, you won't.

By using a Fire tablet as your chosen destination you will not be able to remove DRM if the book has it. DRM-free books will work however. You need to have an e-ink Kindle device as the target in order to get a file that will allow removal of DRM.

Lower maximum image quality is one of the downsides of using Download & Transfer. It does not usually matter for novels, but can be noticeable for things like comics and manga. Changing to a different model of Kindle (or Fire tablet) will not change that. Except for the very oldest Kindle models, the same file will be delivered regardless of chosen device.

Most comics can still be downloaded using the latest version of Kindle for PC in a format that can be de-DRMed and with higher resolution than Download & Transfer supports. Most other types of books will download in that app in KFX format with unbroken DRM. Comics are one of the exceptions.

alehel commented 1 year ago

@j-howell ah, yes. I see. I wasn't aware of this difference. Thanks.

chocolatechipcats commented 1 year ago

According to users in the MobileRead thread, "download and transfer" is now blocked for Kindle Unlimited borrows. I don't know if this indicates they plan to add more restrictions.

VorpalQ commented 1 year ago

According to users in the MobileRead thread, "download and transfer" is now blocked for Kindle Unlimited borrows. I don't know if this indicates they plan to add more restrictions.

I just tested this with a regular fiction book published today on KU. I am able to still 'download and transfer' and copy it to my Kindle. I think more information is needed on this. Is it a specific type of book perhaps? (Comics or Manga maybe?)

federicorosso1993 commented 1 year ago

I know that as a kindle 4 user I'm unable to download some Manga/comics (but they are the one that are not with KFX so I can download and remove drm with the last Kindle pc version)

chocolatechipcats commented 1 year ago

According to users in the MobileRead thread, "download and transfer" is now blocked for Kindle Unlimited borrows. I don't know if this indicates they plan to add more restrictions.

I just tested this with a regular fiction book published today on KU. I am able to still 'download and transfer' and copy it to my Kindle. I think more information is needed on this. Is it a specific type of book perhaps? (Comics or Manga maybe?)

Did purchase it, or borrow? That's something I specifically asked about (no Kindle or Prime to test myself); it applies specifically to borrows (for now).

VorpalQ commented 1 year ago

Okay, I just checked again with a non purchased KU book (borrowed). Still works. Is it possible this is a limited roll out in certain countries only perhaps?

On January 12, 2023 4:07:13 p.m. CST, chocolatechipcats @.***> wrote:

According to users in the MobileRead thread, "download and transfer" is now blocked for Kindle Unlimited borrows. I don't know if this indicates they plan to add more restrictions.

I just tested this with a regular fiction book published today on KU. I am able to still 'download and transfer' and copy it to my Kindle. I think more information is needed on this. Is it a specific type of book perhaps? (Comics or Manga maybe?)

That's something I specifically asked about (no Kindle or Prime to test myself); it applies specifically to borrows (for now).

-- Reply to this email directly or view it on GitHub: https://github.com/noDRM/DeDRM_tools/issues/240#issuecomment-1381046065 You are receiving this because you commented.

Message ID: @.***>

j-howell commented 1 year ago

I can still use Download & Transfer for Kindle Unlimited borrows. Only a few people have reported it being blocked so far. I suspect that it is being rolled out gradually to reduce the impact on customer service.

ElleKayEm commented 1 year ago

Or maybe older Kindles that don't do azw3 and these aren't available in mobi? (haven't looked at the MobileRead thread yet)

chocolatechipcats commented 1 year ago

This post points out Amazon help page stating 'download and transfer' unavailable for borrows: https://www.mobileread.com/forums/showpost.php?p=4288991&postcount=286

ElleKayEm commented 1 year ago

Well, that would be good for authors in KU since they only get paid by the percent of the book you read.

trulow commented 1 year ago

I can still use Download & Transfer for Kindle Unlimited borrows. Only a few people have reported it being blocked so far. I suspect that it is being rolled out gradually to reduce the impact on customer service.

Seems like the limitation is primarily for books published on or after Jan 3, 2023

chocolatechipcats commented 1 year ago

I can still use Download & Transfer for Kindle Unlimited borrows. Only a few people have reported it being blocked so far. I suspect that it is being rolled out gradually to reduce the impact on customer service.

Seems like the limitation is primarily for books published on or after Jan 3, 2023

Tested KU borrows from before and after that date? The link posted above indicates that borrows unavailable for transfer regardless of date. The January 3 cutoff seems to be for any book being downloadable on K4PC 1.17.

ElleKayEm commented 1 year ago

Yes, no KU borrows at all for download & transfer now. (Which is fine. Amazon should have done that for authors all along.) Jan 3 pub date and later purchased books can't be downloaded with any of the older Kindle for PC/Mac -- need to do download & transfer to get in decryptable format. ETA: unless it's a book that doesn't come in KFX at all.