search

noDRM / DeDRM_tools

DeDRM tools for ebooks
7.16k stars 321 forks source link

hardened DRM #20

Closed Sz528716 closed 2 years ago

Sz528716 commented 2 years ago

Question / bug report

Hi. Can anyone extend this code to remove hardened DRM of ACSM files? I can support him to works on it. Thanks

Which version of Calibre are you running?

5.32

Which version of the DeDRM plugin are you running?

v10

If applicable, which version of the Kindle software are you running?

No response

Log output

No response

ElleKayEm commented 2 years ago

As a warning to others, what ebookstore have you found using the hardened DRM?

Sz528716 commented 2 years ago

ProQuest eBooks have hardened DRMs from 2 days ago

ElleKayEm commented 2 years ago

Ok, thanks. At least it isn't on purchased books in that case.

Sz528716 commented 2 years ago

Prior to this update, this code was able to remove DRM with no issue

ElleKayEm commented 2 years ago

I see, they switched from the older Adobe DRM to the newer hardened Adobe DRM. But, again, glad to see it isn't an ebookstore.

noDRM commented 2 years ago

I did look into a book with this new DRM a while ago, but I was unable to figure out a way to remove the DRM. Would probably take a fairly long time to debug / analyze ADE to actually figure out how this new DRM works ...

Do they actually enforce the new DRM (meaning, you cannot download / open books with ADE 2.0), or do they just use it if available? Given that this seems to be a book from a library, can you try lending another book and then open the ACSM file with ADE 2.0? Or did you already do that for this one?

Sz528716 commented 2 years ago

Thanks. I tried many books. I can not open them with ADE 2.0, but there is no problem with with ADE 3 or 4.

noDRM commented 2 years ago

Yeah, if ADE2 doesn't open them at all but ADE3 does then they seem to be actually enforcing the new DRM. That's unlikely to be supported in the near future, unless someone figures out how it works.

balthazoro commented 2 years ago

Oh geez I've just been banging my head against a couple of files downloaded from ProQuest for a couple hours now, without realizing that ProQuest had switched to a hardened DRM. Shame that it may not be supported. I hope some folks are able to sort it out. Anyways here is the debug for calibre 5.34 (I also tried in 4.23 with the olde deDRM 6.8.1 and got the same results):

calibre Debug log calibre 5.34 embedded-python: True is64bit: True macOS-10.16-x86_64-i386-64bit Darwin ('64bit', '') ('Darwin', '21.2.0', 'Darwin Kernel Version 21.2.0: Sun Nov 28 20:28:54 PST 2021; root:xnu-8019.61.5~1/RELEASE_X86_64') Python 3.8.5 OSX: ('10.16', ('', '', ''), 'x86_64') Interface language: None Successfully initialized third party plugins: DeACSM (0, 0, 15) && Gather KFX-ZIP (from KFX Input) (1, 46, 0) && DeDRM (10, 0, 2) && Package KFX (from KFX Input) (1, 46, 0) && KFX metadata reader (from KFX Input) (1, 46, 0) && KFX Input (1, 46, 0) && Set KFX metadata (from KFX Output) (1, 58, 0) && KFX Output (1, 58, 0) calibre 5.34 embedded-python: True is64bit: True macOS-10.16-x86_64-i386-64bit Darwin ('64bit', '') ('Darwin', '21.2.0', 'Darwin Kernel Version 21.2.0: Sun Nov 28 20:28:54 PST 2021; root:xnu-8019.61.5~1/RELEASE_X86_64') Python 3.8.5 OSX: ('10.16', ('', '', ''), 'x86_64') Interface language: None Successfully initialized third party plugins: DeACSM (0, 0, 15) && Gather KFX-ZIP (from KFX Input) (1, 46, 0) && DeDRM (10, 0, 2) && Package KFX (from KFX Input) (1, 46, 0) && KFX metadata reader (from KFX Input) (1, 46, 0) && KFX Input (1, 46, 0) && Set KFX metadata (from KFX Output) (1, 58, 0) && KFX Output (1, 58, 0) devicePixelRatio: 2.0 logicalDpi: 72.0 x 72.0 physicalDpi: 108.50000162950651 x 107.50000161448803 Using calibre Qt style: True [0.00] Starting up... [0.02] Showing splash screen... [0.24] splash screen shown [0.24] Initializing db... [0.24] db initialized [0.24] Constructing main UI... [0.78] main UI initialized... [0.78] Hiding splash screen Starting QuickView DeDRM v10.0.2: Trying to decrypt xxxxxxxxxxxx.pdf DeDRM v10.0.2: Undersea Geopolitics.pdf is a PDF ebook for UUID 45bb3646-9f0b-410a-9aab-1cba8bbfa2a4 DeDRM v10.0.2: Looking for new default Adobe Digital Editions Keys after 0.0 seconds DeDRM v10.0.2: Trying a new default key This seems to be an Adobe ADEPT PDF with Adobe's new DRM This DRM cannot be removed yet. Try getting your distributor to give you a new ACSM file, then open that in an old version of ADE (2.0). If your book distributor is not enforcing the new DRM yet, this will give you a copy with the old DRM. DeDRM v10.0.2: Exception when decrypting after 0.1 seconds Traceback (most recent call last): File "calibre_plugins.dedrm.init", line 712, in PDFDecrypt result = ineptpdf.decryptBook(userkey, path_to_ebook, of.name) File "calibre_plugins.dedrm.ineptpdf", line 2322, in decryptBook serializer = PDFSerializer(inf, userkey, inept) File "calibre_plugins.dedrm.ineptpdf", line 2128, in init doc.initialize(userkey, inept) File "calibre_plugins.dedrm.ineptpdf", line 1504, in initialize return self.initialize_ebx_inept(password, docid, param) File "calibre_plugins.dedrm.ineptpdf", line 1708, in initialize_ebx_inept raise ADEPTNewVersionError("Book uses new ADEPT encryption") calibre_plugins.dedrm.ineptpdf.ADEPTNewVersionError: Book uses new ADEPT encryption DeDRM v10.0.2: Failed to decrypt with new default key after 0.1 seconds DeDRM v10.0.2: Ultimately failed to decrypt after 0.1 seconds. Read the FAQs at Harper's repository: https://github.com/apprenticeharper/DeDRM_tools/blob/master/FAQs.md Running file type plugin DeDRM failed with traceback: Traceback (most recent call last): File "calibre/customize/ui.py", line 175, in _run_filetype_plugins File "calibre_plugins.dedrm.init", line 895, in run decrypted_ebook = self.PDFDecrypt(path_to_ebook) File "calibre_plugins.dedrm.init", line 770, in PDFDecrypt raise DeDRMError("{0} v{1}: Ultimately failed to decrypt after {2:.1f} seconds. Read the FAQs at Harper's repository: https://github.com/apprenticeharper/DeDRM_tools/blob/master/FAQs.md".format(PLUGIN_NAME, PLUGIN_VERSION, time.time()-self.starttime)) calibre_plugins.dedrm.DeDRMError: DeDRM v10.0.2: Ultimately failed to decrypt after 0.1 seconds. Read the FAQs at Harper's repository: https://github.com/apprenticeharper/DeDRM_tools/blob/master/FAQs.md Traceback (most recent call last): File "calibre/customize/ui.py", line 432, in get_file_type_metadata File "calibre/customize/builtins.py", line 318, in get_metadata File "calibre/ebooks/metadata/pdf.py", line 124, in get_metadata ValueError: Could not read info dict from PDF Syntax Error: Couldn't find the 'EBX_HANDLER' security handler pdfinfo errored out with return code: 1 [2.87] splash screen hidden [2.87] Started up in 2.87 seconds with 0 books

Debugging DND event text/uri-list 87 b'file:///Users/xxxxxxxxxxxx

Firefox rurl: None None URLS: ['file:///Users/xxxxxxxxxxxx Paths: ['/Users/xxxxxxxxxxxx Extensions: frozenset({'pdf'}) Added xxxxxxxxxxx to db in: 0.0 Added 1 books in 0.7 seconds

noDRM commented 2 years ago

Yep, the main error message from that log is 

This seems to be an Adobe ADEPT PDF with Adobe's new DRM
This DRM cannot be removed yet.
Try getting your distributor to give you a new ACSM file, then open that in an old version of ADE (2.0).
If your book distributor is not enforcing the new DRM yet, this will give you a copy with the old DRM.

which means that the plugin detected the new DRM.

Right now there's no way to remove that. I've been looking into that from time to time, but so far I was unable to figure out the solution.

balthazoro commented 2 years ago

Yep, the main error message from that log is

This seems to be an Adobe ADEPT PDF with Adobe's new DRM
This DRM cannot be removed yet.
Try getting your distributor to give you a new ACSM file, then open that in an old version of ADE (2.0).
If your book distributor is not enforcing the new DRM yet, this will give you a copy with the old DRM.

which means that the plugin detected the new DRM.

Right now there's no way to remove that. I've been looking into that from time to time, but so far I was unable to figure out the solution.

Understood. I will keep fingers crossed and on the look out for any solutions. I figure that there may be a lot more people running into this going forward, since calibre + DRM removal is used by so many academics and researchers, fields where ProQuest ebooks and the like end up being a significant resource. Hopefully there will be a critical mass on solving for this new DRM before long.

jbsinger1970 commented 2 years ago

This has been a very helpful thread.

I just upgraded to Calibre 5.34 from 4.23. I tried installing deDRM 10 and 7.2.1 and got the same error message: ERROR: Unhandled exception: InvalidPlugin:The plugin in 'C:\Users\jonat\Downloads\DeDRM_tools_7.2.1.zip' is invalid. It does not contain a top-level init.py file. Anyone know what that means?

ZolaLa9 commented 2 years ago

From the FAQ: You are trying to add the tools archive (e.g. DeDRM_tools_6.8.0.zip) instead of the plugin. The tools archive is not the plugin. It is a collection of DRM removal tools which includes the plugin. You must unzip the archive, and install the calibre plugin DeDRM_plugin.zip from a folder called DeDRM_calibre_plugin in the unzipped archive.

ElleKayEm commented 2 years ago

FYI, there is no DeDRM_calibre_plugin folder inside the archive anymore. So just unzip DeDRM_tools_7.2.1.zip and load DeDRM_plugin.zip.

jbsinger1970 commented 2 years ago

Ok. I'm an idiot. Of course. Thanks.

johnhutnut commented 2 years ago

Adobe Content Server 5 (ACS 5) DRM. can not work，

johnhutnut commented 2 years ago

calibre, version 5.38.0 : ModuleNotFoundError:No module named '__version'

calibre 5.38 [64bit] embedded-python: True is64bit: True Windows-10-10.0.22000 Windows ('64bit', 'WindowsPE') ('Windows', '10', '10.0.22000') Python 3.8.5 Windows: ('10', '10.0.22000', '', 'Multiprocessor Free') Interface language: zh_CN Successfully initialized third party plugins: DeDRM (10, 0, 2) && Obok DeDRM (10, 0, 0) Traceback (most recent call last): File "calibre\gui2\preferences\plugins.py", line 326, in add_plugin File "calibre\customize\ui.py", line 476, in add_plugin File "calibre\customize\ui.py", line 64, in load_plugin File "calibre\customize\zipplugin.py", line 290, in load File "polyglot\builtins.py", line 123, in reload File "importlib__init.py", line 169, in reload File "", line 604, in _exec File "calibre\customize\zipplugin.py", line 191, in exec_module File "calibre_plugins.dedrm.init", line 97, in ModuleNotFoundError: No module named 'version'

ElleKayEm commented 2 years ago

@johnhutnut Are you trying to use this: https://github.com/noDRM/DeDRM_tools/pull/48

johnhutnut commented 2 years ago

@johnhutnut Are you trying to use this: #48

yes

ElleKayEm commented 2 years ago

This comment might help: https://github.com/noDRM/DeDRM_tools/pull/48#issuecomment-1059044883

johnhutnut commented 2 years ago

This comment might help: #48 (comment)

but i install https://github.com/noDRM/DeDRM_tools 10.0.2 work fine， the 10.0.2 can not decypt Adobe Content Server 5 (ACS 5) DRM. of course 10.0.2 can decypt Adobe Content Server 4 (ACS 4) DRM

johnhutnut commented 2 years ago

Question / bug report

Hi. Can anyone extend this code to remove hardened DRM of ACSM files? I can support him to works on it. Thanks my email ab1307711@hotmail.com

Which version of Calibre are you running?

Which version of the DeDRM plugin are you running?

calibre, version 5.38.0 ModuleNotFoundError:No module named '__version'

calibre 5.38 [64bit] embedded-python: True is64bit: True Windows-10-10.0.22000 Windows ('64bit', 'WindowsPE') ('Windows', '10', '10.0.22000') Python 3.8.5 Windows: ('10', '10.0.22000', '', 'Multiprocessor Free') Interface language: zh_CN Successfully initialized third party plugins: DeDRM (10, 0, 2) && Obok DeDRM (10, 0, 0) Traceback (most recent call last): File "calibre\gui2\preferences\plugins.py", line 326, in add_plugin File "calibre\customize\ui.py", line 476, in add_plugin File "calibre\customize\ui.py", line 64, in load_plugin File "calibre\customize\zipplugin.py", line 290, in load File "polyglot\builtins.py", line 123, in reload File "importlib__init.py", line 169, in reload File "", line 604, in _exec File "calibre\customize\zipplugin.py", line 191, in exec_module File "calibre_plugins.dedrm.init", line 97, in ModuleNotFoundError: No module named 'version'

v10

If applicable, which version of the Kindle software are you running?

No response

Log output

No response

ElleKayEm commented 2 years ago

Looks like others have had success with the new pull request.

noDRM commented 2 years ago

Please keep support requests for this not-yet-merged PR over at #48 , there's no need to discuss these issues in multiple places.

yuan0qi commented 2 years ago

I'm encountering a similar issue:

DeDRM v10.0.2: Trying to decrypt *****.pdf
DeDRM v10.0.2: *****.pdf is a PDF ebook with encryption EBX_HANDLER
DeDRM v10.0.2: *****.pdf is a PDF ebook (EBX) for UUID *****
DeDRM v10.0.2: Trying UUID-matched encryption key *****
error writing pdf: Invalid object number: objid=766
Traceback (most recent call last):
  File "/Users/yuanqi/Library/Preferences/calibre/plugins/DeDRM.zip/ineptpdf.py", line 1807, in getobj
    obj = objs[i]
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/yuanqi/Library/Preferences/calibre/plugins/DeDRM.zip/ineptpdf.py", line 2277, in decryptBook
    serializer.dump(outf)
  File "/Users/yuanqi/Library/Preferences/calibre/plugins/DeDRM.zip/ineptpdf.py", line 2105, in dump
    obj = doc.getobj(objid)
  File "/Users/yuanqi/Library/Preferences/calibre/plugins/DeDRM.zip/ineptpdf.py", line 1809, in getobj
    raise PDFSyntaxError('Invalid object number: objid=%r' % (objid))
ineptpdf.PDFSyntaxError: Invalid object number: objid=766
DeDRM v10.0.2: Trying encryption key *****
error writing pdf: Invalid object number: objid=766
Traceback (most recent call last):
  File "/Users/yuanqi/Library/Preferences/calibre/plugins/DeDRM.zip/ineptpdf.py", line 1807, in getobj
    obj = objs[i]
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/yuanqi/Library/Preferences/calibre/plugins/DeDRM.zip/ineptpdf.py", line 2277, in decryptBook
    serializer.dump(outf)
  File "/Users/yuanqi/Library/Preferences/calibre/plugins/DeDRM.zip/ineptpdf.py", line 2105, in dump
    obj = doc.getobj(objid)
  File "/Users/yuanqi/Library/Preferences/calibre/plugins/DeDRM.zip/ineptpdf.py", line 1809, in getobj
    raise PDFSyntaxError('Invalid object number: objid=%r' % (objid))
ineptpdf.PDFSyntaxError: Invalid object number: objid=766
DeDRM v10.0.2: Failed to decrypt with key ***** after 1.3 seconds
DeDRM v10.0.2: Looking for new default Adobe Digital Editions Keys after 1.3 seconds
DeDRM v10.0.2: Trying a new default key
error writing pdf: Invalid object number: objid=766
Traceback (most recent call last):
  File "/Users/yuanqi/Library/Preferences/calibre/plugins/DeDRM.zip/ineptpdf.py", line 1807, in getobj
    obj = objs[i]
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/yuanqi/Library/Preferences/calibre/plugins/DeDRM.zip/ineptpdf.py", line 2277, in decryptBook
    serializer.dump(outf)
  File "/Users/yuanqi/Library/Preferences/calibre/plugins/DeDRM.zip/ineptpdf.py", line 2105, in dump
    obj = doc.getobj(objid)
  File "/Users/yuanqi/Library/Preferences/calibre/plugins/DeDRM.zip/ineptpdf.py", line 1809, in getobj
    raise PDFSyntaxError('Invalid object number: objid=%r' % (objid))
ineptpdf.PDFSyntaxError: Invalid object number: objid=766
DeDRM v10.0.2: Failed to decrypt with new default key after 2.1 seconds
DeDRM v10.0.2: Finished after 2.1 seconds
Traceback (most recent call last):
  File "calibre/customize/ui.py", line 432, in get_file_type_metadata
  File "calibre/customize/builtins.py", line 315, in get_metadata
  File "calibre/ebooks/metadata/pdf.py", line 124, in get_metadata
ValueError: Could not read info dict from PDF
Syntax Error: Couldn't find the 'EBX_HANDLER' security handler
Added ***** to db in: 0.0
Added 1 books in 3.2 seconds

Oh geez I've just been banging my head against a couple of files downloaded from ProQuest for a couple hours now, without realizing that ProQuest had switched to a hardened DRM. Shame that it may not be supported. I hope some folks are able to sort it out. Anyways here is the debug for calibre 5.34 (I also tried in 4.23 with the olde deDRM 6.8.1 and got the same results):

ElleKayEm commented 2 years ago

@yuan0qi Have you tried with this: https://github.com/noDRM/DeDRM_tools/pull/48

yuan0qi commented 2 years ago

@yuan0qi Have you tried with this: #48

Yes, I am using the plugin from the hardened DRM branch.

noDRM commented 2 years ago

As for the __version issue, there's a new test version over at #48 that should have some more verbose logging to help debug why that shows up.

As for the error writing pdf: Invalid object number: objid=766 issue, that looks like an issue with all the PDF parsing code. That's probably something that someone with more knowledge about the PDF format needs to take a look at ...