Fix legacy crypto support for OpenSSL 3

[Leseratte10]

There seem to be two bugs in the existing code (as reported in #60 )

A) OSSL_PROVIDER_available indicates if a provider is loaded - if it is available to be used. Not if it is installed, as in, available to be loaded. That means, it will always be false as in the default config (at least on Ubuntu 22.04) it's not loaded. You first need to (try to) load the legacy provider, then check if that was successful.

Quoting from the OpenSSL commit message that introduced this function:

*) Introduced a new function, OSSL_PROVIDER_available(), which can be used
   to check if a named provider is loaded and available.  When called, it
   will also activate all fallback providers if such are still present.
   [Richard Levitte]

B) Once you load any provider (in this case, "legacy"), OpenSSL no longer loads the default provider with all the non-legacy code. You need to load both, one after the other. This bug didn't affect anything in 1.3.0 because due to bug A), the code never actually tried to load the legacy module, at least in my tests.

[wbond]

I had run into issues with OpenSSL 3 on macOS. I think I also made a VMWare image of 22.04 for testing. Either way, we should make sure this config is tested via CI.

[Leseratte10]

Haven't written many tests before so I'm unsure how to do that.

Sure, I could write a test case that tries to decrypt something that's only supported by the legacy provider, but that test would then fail if you actually had an environment where the legacy provider is not available.

[wbond]

I think at this point some info about the legacy module being present (maybe results of ls) and how legacy functions aren’t working would be a good start. More or less a straightforward way to reproduce.

Not sure if any of the CI providers have Ubuntu 22.04 yet that we can add for testing, but that should come after a reproducer.

[wbond]

Right now both before and after this PR all of CI passes, so we just need to make sure we have a failing test that changes to successful.

[Leseratte10]

Okay, should now be done. On the master branch (the source of this PR) is my bugfix and a successful test case. On the master_pre_fix you see just the new test case, which fails on Ubuntu 22.04 and on MacOS 10.15 (to be expected, you said you ran into issues with OpenSSL on MacOS).

I also added another CI target for Ubuntu 22.04 with Python 2, just in case.

The test case tries to load the "legacy" module the proper way (just load and see if it works, rather than trying to check that before), and if that was successful (= the test case was able to load the legacy module), but libcrypto_legacy_support is False; then the test fails.

[Leseratte10]

Anything I still need to do to get this merged, or is this just waiting for review? Wondering if it's worth the effort to make my application use my fixed fork of oscrypto until it is, or if it's simpler to just wait for a new release.

EDIT: I have embedded a fork of oscrypto into my application with exactly this patch, and multiple people confirmed that this fixes the issue with legacy crypto on Ubuntu 22.04, Fedora 36 and Debian unstable.

[Leseratte10]

Any update to this PR, @wbond ? If this is not getting updated I'll have to fork or monkey-patch oscrypto for my application (which I'd rather avoid), but seeing that there has been no response in the last 2 months I might have to do that ... Or is anything still missing before this PR can be merged and put into a new release?

[ourichermath]

What's going on with this? This had a PR back on 6 May (coming up on 3 months ago) and nothing's happened. Does Leseratte10 need to do anything else? Right at the bottom of this thread, Github appears to be saying the submission has passed all checks.

[wbond]

oscrypto just hasn’t bubbled up to the top of my open source priority queue yet.

In the meantime, feel free to test this PR out in your stack.

[Leseratte10]

Thanks for your response. I'm going to use my own fork in the meantime then until this is merged and included in a new release.

[fizzlifax2]

Yea there is also a Problem installing the Plugin in Calibre with the following messages - (I downloaded the Zip-File from the rep and tried it ) - I have Ubuntu 20.04.4 LTS ################################# calibre, version 4.99.4 FEHLER: Unbehandelter Ausnahmefehler: InvalidPlugin:Initialisierung der Erweiterung Traceback (most recent call last): File "/usr/lib/calibre/calibre/customize/ui.py", line 665, in initialize_plugin p.initialize() File "calibre_plugins.deacsm.init", line 226, in initialize ModuleNotFoundError: No module named 'libadobe' fehlgeschlagen. Rückverfolgung: Traceback (most recent call last): File "/usr/lib/calibre/calibre/customize/ui.py", line 665, in initialize_plugin p.initialize() File "calibre_plugins.deacsm.init", line 226, in initialize ModuleNotFoundError: No module named 'libadobe'

calibre 4.99.4 embedded-python: False is64bit: True Linux-5.15.0-43-generic-x86_64-with-glibc2.29 Linux ('64bit', 'ELF') ('Linux', '5.15.0-43-generic', '#46~20.04.1-Ubuntu SMP Thu Jul 14 15:20:17 UTC 2022') Python 3.8.10 Interface language: de Traceback (most recent call last): File "/usr/lib/calibre/calibre/customize/ui.py", line 665, in initialize_plugin p.initialize() File "calibre_plugins.deacsm.init", line 226, in initialize ModuleNotFoundError: No module named 'libadobe'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/lib/calibre/calibre/gui2/preferences/plugins.py", line 319, in add_plugin plugin = add_plugin(path) File "/usr/lib/calibre/calibre/customize/ui.py", line 476, in add_plugin plugin = initialize_plugin(plugin, path_to_zip_file) File "/usr/lib/calibre/calibre/customize/ui.py", line 670, in initializeplugin raise InvalidPlugin((('Initialization of plugin %s failed with traceback:') calibre.customize.InvalidPlugin: Initialisierung der Erweiterung Traceback (most recent call last): File "/usr/lib/calibre/calibre/customize/ui.py", line 665, in initialize_plugin p.initialize() File "calibre_plugins.deacsm.init", line 226, in initialize ModuleNotFoundError: No module named 'libadobe' fehlgeschlagen. Rückverfolgung: Traceback (most recent call last): File "/usr/lib/calibre/calibre/customize/ui.py", line 665, in initialize_plugin p.initialize() File "calibre_plugins.deacsm.init", line 226, in initialize ModuleNotFoundError: No module named 'libadobe'

#################################################### And there is also an error if I try to install it from the available extensions directly like: ######################################################## calibre, version 4.99.4 FEHLER: Erweiterungsinstallation fehlgeschlagen: Während der Installation der Erweiterung ist ein Problem aufgetreten. Diese Erweiterung wird nun entfernt. Bitte veröffentlichen Sie die folgende detaillierte Fehlermeldung im Diskussionsforum dieser Erweiterung und starten Sie Calibre neu.

Traceback (most recent call last): File "/usr/lib/calibre/calibre/customize/ui.py", line 665, in initialize_plugin p.initialize() File "calibre_plugins.deacsm.init", line 226, in initialize ModuleNotFoundError: No module named 'libadobe'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/lib/calibre/calibre/gui2/dialogs/plugin_updater.py", line 725, in _install_clicked plugin = add_plugin(zip_path) File "/usr/lib/calibre/calibre/customize/ui.py", line 476, in add_plugin plugin = initialize_plugin(plugin, path_to_zip_file) File "/usr/lib/calibre/calibre/customize/ui.py", line 670, in initializeplugin raise InvalidPlugin((('Initialization of plugin %s failed with traceback:') calibre.customize.InvalidPlugin: Initialisierung der Erweiterung Traceback (most recent call last): File "/usr/lib/calibre/calibre/customize/ui.py", line 665, in initialize_plugin p.initialize() File "calibre_plugins.deacsm.init", line 226, in initialize ModuleNotFoundError: No module named 'libadobe' fehlgeschlagen. Rückverfolgung: Traceback (most recent call last): File "/usr/lib/calibre/calibre/customize/ui.py", line 665, in initialize_plugin p.initialize() File "calibre_plugins.deacsm.init", line 226, in initialize ModuleNotFoundError: No module named 'libadobe' ####################################

Thanks a lot in advance!

[Leseratte10]

This doesn't really fit into this bug report since it has nothing to do with oscrypto. The bug mentioned here is fixed with the new plugin (which uses a fork of oscrypto). Looking at the version number you're using an old Calibre-5 beta - if you're on Ubuntu, don't install from the repo but directly from calibre-ebook.com. For more questions about the plugin please use my repo and not this PR.

[wbond]

Sorry for the delay in getting to this @Leseratte10, but I'm merging this now. Thanks for working on it and your patience!

[Leseratte10]

Thanks for getting this merged; is there any plans to release this as 1.3.1 or 1.4.0 on PyPi with that fix? I see that "pip install oscrypto" currently still gets you the old broken version of the library.