Using PKCS#7 OpenSSL API

Muzosh commented 1 year ago

Hello!

I'm having issues running timestamping server with oqsprovider.

I have this Dockerfile:

FROM bitnami/minideb:bullseye

RUN install_packages \
    ca-certificates curl libssl-dev cmake libcivetweb-dev gcc g++ make git build-essential cmake ninja-build checkinstall zlib1g-dev wget

# OPENSSL@3
WORKDIR /root/opensslv3
RUN wget https://github.com/openssl/openssl/releases/download/openssl-3.1.2/openssl-3.1.2.tar.gz \
    && tar --strip-components=1 -zxvf openssl-3.1.2.tar.gz \
    && ./config \
    && make \
    && make install \
    && ldconfig

# OQS-PROVIDER
WORKDIR /root/oqsprovider
RUN git clone --branch 0.5.1 --depth 1 https://github.com/open-quantum-safe/oqs-provider.git .\
    && ./scripts/fullbuild.sh
RUN sed -i 's/default = default_sect/default = default_sect\noqsprovider = oqsprovider_sect\n\n\[oqsprovider_sect\]\nactivate = 1/g' /usr/local/ssl/openssl.cnf && sed -i 's/# activate = 1/activate = 1/g' /usr/local/ssl/openssl.cnf
RUN cp /root/oqsprovider/_build/lib/oqsprovider.so /usr/local/lib/ossl-modules/oqsprovider.so

# UTS-SERVER
WORKDIR /root/uts-server
RUN wget https://codeload.github.com/kakwa/uts-server/tar.gz/refs/tags/0.2.1 \
    && tar --strip-components=1 -zxvf 0.2.1\
    && cmake . \
    && make

WORKDIR /opt/uts-server

RUN groupadd -r uts-server \
    && useradd -r -g uts-server uts-server \
    && mkdir /etc/uts-server \
    && chown uts-server:uts-server /etc/uts-server

VOLUME /etc/uts-server

COPY --from=builder /root/uts-server .

EXPOSE 2020

ENTRYPOINT ["./uts-server"]

I can verify that oqsprovider is installed:

root@48d18a97c59f:/opt/uts-server# openssl list -providers
Providers:
  default
    name: OpenSSL Default Provider
    version: 3.1.2
    status: active
  oqsprovider
    name: OpenSSL OQS Provider
    version: 0.5.1
    status: active

All certificates and private keys I'm feeding it, are Dilithium5. Console openssl with oqsprovider can work with them without any issues.

In the software (using OpenSSL API), I was having issues with populating config values, proper keys, etc. But now, I'm stuck at function TS_RESP_create_response, which trows multiple OPENSSL errors (the software obtains them by repeatedly calling ERR_get_error() and printing them):

LOG_DEBUG. : OpenSSL exception: 'error:10800094:PKCS7 routines::signing not supported for this key type'
LOG_DEBUG  : OpenSSL exception: 'error:17800076:time stamp routines::pkcs7 add signature error'
LOG_DEBUG  : OpenSSL exception: 'error:17800079:time stamp routines::response setup error'

I tried to fix that by running this code before-hand (with #include <openssl/provider.h>):

OSSL_PROVIDER *oqsprovider;

oqsprovider = OSSL_PROVIDER_load(NULL, "oqsprovider");
if (oqsprovider == NULL) {
    printf("Failed to load Legacy provider\n");
    exit(EXIT_FAILURE);
}

if (ERR_get_error() == 0){
    uts_logger(ct, LOG_DEBUG, "no openssl error");
}

printf("Successfully loaded provider\n");

, which runs successfully (I see both no openssl error and Successfully loaded provider in the output. But still, this does not solve the issue and I'm still getting error:10800094:PKCS7 routines::signing not supported for this key type error.

Environment (please complete the following information):

OS: docker minideb
OpenSSL version: 3.2.1
oqsprovider version: 0.5.1

Please run the following commands to obtain the version information:

For OpenSSL: openssl version: OpenSSL 3.1.2 1 Aug 2023 (Library: OpenSSL 3.1.2 1 Aug 2023)
For oqsprovider: openssl list -providers: already posted here

Here is the certificate of a public key, associated with the private key used for signing.

I won't post private key here even if it is just a testing value, but I checked it's ASN1 structure and it's just regular PrivateKeyInfo with 1.3.6.1.4.1.2.267.7.8.7 algorithm and 7456B octet string. Also, as I said, console openssl does not have issues working with this key.

Is there something I'm missing? Is this even an issue related to OQSProvider?

baentsch commented 1 year ago

Is there something I'm missing? Is this even an issue related to OQSProvider?

"Probably Yes" and "Possibly No" in that order:

You are running

OpenSSL 3.1.2 1 Aug 2023 (Library: OpenSSL 3.1.2 1 Aug 2023)

This OpenSSL version cannot handle OQS signatures in TLS operations as per documentation.

That said, I cannot be entirely certain that this is the issue here without seeing the full OpenSSL stack trace leading to this error. What you could do to test this out is use OpenSSL3.2 instead and see whether the error persists.

Muzosh commented 1 year ago

Thank you for looking into this. Unfortunately, bumping the OpenSSL version to ~3.1.2~ 3.2.0-alpha1 did not help.

Is there any other way for me to log openssl errors in more detail?

Currently, I do it as such:

while ((err_code = ERR_get_error()))
{
    ERR_load_TS_strings();
    uts_logger(ct, LOG_ERR, "---\nERR_error_string: '%s'\nERR_reason_error_string: '%s'\nERR_lib_error_string: '%s'\n---\n",
                ERR_error_string(err_code, NULL), ERR_reason_error_string(err_code), ERR_lib_error_string(err_code));
}

I'm not really familiar with OpenSSL C API and I can't find any documentation for it.

baentsch commented 1 year ago

Unfortunately, bumping the OpenSSL version to 3.1.2 did not help.

You need to bump to 3.2.

Any 3.1 variant doesn't have all the required capabilities.

I'm not really familiar with OpenSSL C API and I can't find any documentation for it.

Start from https://www.openssl.org/docs/manmaster/man3/ERR_print_errors.html

Muzosh commented 1 year ago

Unfortunately, bumping the OpenSSL version to 3.1.2 did not help.

My mistake, I bumped it to 3.2, I just messed up my comment here on GitHub :)

baentsch commented 1 year ago

Unfortunately, bumping the OpenSSL version to 3.1.2 did not help.

My mistake, I bumped it to 3.2, I just messed up my comment here on GitHub :)

OK -- then that isn't the issue. The API you're calling TS_RESP_create_response: What is that doing? I didn't find any documentation about it. I guess what would be best is code to allow us to reproduce the problem.

baentsch commented 1 year ago

Just did a quick API check: This seems to activate the PKCS7 logic of OpenSSL. I am unsure as to whether this is fully provider enabled. At least, no reference to providers are made there -- unlike in the code for CMS, which does work OK with provider-based signature algorithms. Maybe @levitte can provide insight?

Muzosh commented 1 year ago

I am unsure as to whether this is fully provider enabled.

Is this related only to oqsprovider? I.e. would using OQS-OpenSSL@1.1 fork solve this (I'm aware it is discontinued)?

Muzosh commented 1 year ago

So I tried it with OQS-OpenSSL@1.1.1-stable fork and I get two more detailed errors:

error:1012F040:elliptic curve routines:pkey_oqs_ctrl:fatal
error:21081093:PKCS7 routines:PKCS7_SIGNER_INFO_set:signing ctrl failure
error:2F088076:time stamp routines:ts_RESP_sign:pkcs7 add signature error
error:2F07A079:time stamp routines:TS_RESP_create_response:response setup error

root@86cedb6eb6d2:~/uts-server# openssl version -a
OpenSSL 1.1.1u  30 May 2023, Open Quantum Safe 2023-07
built on: Tue Sep 12 11:11:38 2023 UTC
platform: linux-aarch64
options:  bn(64,64) rc4(char) des(int) idea(int) blowfish(ptr) 
compiler: gcc -fPIC -pthread -Ioqs/include -Wa,--noexecstack -Wall -O3 -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DVPAES_ASM -DECP_NISTZ256_ASM -DPOLY1305_ASM -DNDEBUG
OPENSSLDIR: "/usr/local/ssl"
ENGINESDIR: "/usr/local/lib/engines-1.1"
Seeding source: os-specific

levitte commented 1 year ago

Unfortunately, PKCS#7 and CMS aren't fully supported for use with providers... yet. I'm currently writing a design for the missing bits. It's too late for them to appear in 3.2, unfortunartely, but I hope to be able to target 3.3.

Muzosh commented 1 year ago

@levitte Thank you for the response! I completely understand. Would you be able to provide at least non-specific timeframe for OpenSSLv3.3 or PKCS7+CMS functionality? Based on available info, I'm guessing Q1-2 of 2024 to start exploring this?

levitte commented 1 year ago

Q2-2024 is a good target to start exploring, yeah. Or if you feel adventurous, have a look at what happens on https://github.com/openssl/openssl

baentsch commented 1 year ago

Unfortunately, PKCS#7 and CMS aren't fully supported for use with providers... yet.

Thanks for the information. CMS does work reasonably well with oqsprovider, though: Which features are missing? Please let me know if there are concrete things where I could be of assistance.

levitte commented 1 year ago

The concrete things that's missing is a replacement mechanism for these ctrls: EVP_PKEY_CTRL_CMS_ENCRYPT, EVP_PKEY_CTRL_CMS_DECRYPT and EVP_PKEY_CTRL_CMS_SIGN. Their intent is really to pass the AlgorithmIdentifier param to the backend (legacy / engine) implementation and having it deal with them in whatever way they see fit. Those ctrls do so by passing the whole CMS-something structure and letting the backend have free reins on it, which isn't at all a supported way of doing things with providers (and in my very frank opinion, is a hack that should never have seen the light of day).

So basically, that's the bit that needs a better replacement to work with providers.

levitte commented 1 year ago

There are similar ctrls for PKCS7, which I believe is what's hitting @Muzosh

baentsch commented 1 year ago

The concrete things that's missing is a replacement mechanism for these ctrls: EVP_PKEY_CTRL_CMS_ENCRYPT, EVP_PKEY_CTRL_CMS_DECRYPT and EVP_PKEY_CTRL_CMS_SIGN.

Understood (for encrypt/decrypt: I never tested that with oqsprovider); but we do test sign/verify: Why does this work if "EVP_PKEY_CTRL_CMS_SIGN" is not properly wired up for providers? Or is my test wrong?

levitte commented 1 year ago

I actually don't know. I would need some test cases that should trigger them, and time.

baentsch commented 4 months ago

While "cleaning up" I came across this issue again, @levitte Any news on this? Would it make sense to move this into an (new or existing?) OpenSSL issue for more people to notice/possibly contribute as it doesn't seem OQS-specific?

levitte commented 4 months ago

Well, I actually have some work items coming up that are related to the ctrls that are mentioned here. They are basically all about passing the AlgorithmIdentifier params back and forth, just that the ctrls are [ahem] quite convoluted. I'm taking the approach that the AlgoID params can be passed through the usual OSSL_PARAM passing in a more general way (we already have this sort of functionality for ciphers, specifically), and it will be up to the provider to react appropriately.

baentsch commented 4 months ago

it will be up to the provider to react appropriately.

This provider's all too eager to react ... :-) Let's see whether it's appropriate... :-/ Seriously, thanks for letting me know. Please touch base when I could try something.

levitte commented 3 months ago

I've finally started specifying how that should work. The base idea is that it should all work through OSSL_PARAM, like I said earlier.

The supporting OSSL_PARAM keys are (hopefully, pending approval) going to be:

"algorithm-id" (macro: ALG_PARAM_ALGORITHM_ID), a gettable, for which the provider implementation is expected to respond with a DER coded AlgorithmIdentifier for the current invocation.
"algorithm-id-params" (macro: ALG_PARAM_ALGORITHM_ID_PARAMS), both a gettable and a settable, to pass AlgorithmIdentifier.parameters, in its exact binary form. (it shouldn't be necessary to have it as a gettable, most applications should find the full AlgorithmIdentifier, i.e. "algorithm-id", more suitable, but we already have a precedent with EVP_CIPHER_param_to_asn1(), which utilizes this OSSL_PARAM key.

That should cover all the intended uses of EVP_PKEY_CTRL_{PKCS7,CMS}_{DECRYPT,ENCRYPT,SIGN} insofar as passing data between the calling application/library and the backend (provider implementation). Anything else, at least on the backend, should simply be the backend reacting to being passed AlgorithmIdentifier.parameters.

levitte commented 3 months ago

There's actually an OpenSSL design document talking about this: https://github.com/openssl/openssl/blob/master/doc/designs/passing-algorithmidentifier-parameters.md

I had forgotten I wrote that...

beldmit commented 3 months ago

@levitte shouldn't we really leave PKCS7 as is and add new features to CMS only?

levitte commented 3 months ago

That would effectively kill any use of PKCS#7 with providers. However, that discussion is better suited in an OpenSSL discussion than here 😉

open-quantum-safe / oqs-provider

Using PKCS#7 OpenSSL API #251