Can't see dstu support after installing package

[gmile]

Hi @muromec!

After installing openssl-dstu package on Ubuntu Trusty, openssl says there's no dstu available (very likely this is due to my limited knowledge, e.g. I'm doing something wrong).

Installation:

sudo apt-get install openssl=1.0.1f-1ubuntu2.7dstu0~trusty

Usage (gost is there, but neither dstu nor uadstu):

root@899fa5af2dc2:/# openssl engine -t gost
(gost) Reference implementation of GOST engine
     [ available ]
root@899fa5af2dc2:/# openssl engine -t dstu
140443847997088:error:25066067:DSO support routines:DLFCN_LOAD:could not load the shared library:dso_dlfcn.c:185:filename(/usr/lib/x86_64-linux-gnu/openssl-1.0.0/engines/libdstu.so): /usr/lib/x86_64-linux-gnu/openssl-1.0.0/engines/libdstu.so: cannot open shared object file: No such file or directory
140443847997088:error:25070067:DSO support routines:DSO_load:could not load the shared library:dso_lib.c:244:
140443847997088:error:260B6084:engine routines:DYNAMIC_LOAD:dso not found:eng_dyn.c:450:
140443847997088:error:2606A074:engine routines:ENGINE_by_id:no such engine:eng_list.c:417:id=dstu
root@899fa5af2dc2:/# openssl engine -t uadstu
140376049747616:error:25066067:DSO support routines:DLFCN_LOAD:could not load the shared library:dso_dlfcn.c:185:filename(/usr/lib/x86_64-linux-gnu/openssl-1.0.0/engines/libuadstu.so): /usr/lib/x86_64-linux-gnu/openssl-1.0.0/engines/libuadstu.so: cannot open shared object file: No such file or directory
140376049747616:error:25070067:DSO support routines:DSO_load:could not load the shared library:dso_lib.c:244:
140376049747616:error:260B6084:engine routines:DYNAMIC_LOAD:dso not found:eng_dyn.c:450:
140376049747616:error:2606A074:engine routines:ENGINE_by_id:no such engine:eng_list.c:417:id=uadstu
root@899fa5af2dc2:/#

What am I doing wrong?

[gmile]

To give more details, here's a contents of the folder where openssl tries to find the engine:

root@899fa5af2dc2:/# ls -lat /usr/lib/x86_64-linux-gnu/openssl-1.0.0/engines/
total 280
drwxr-xr-x 2 root root  4096 Nov 17 15:34 .
drwxr-xr-x 3 root root  4096 Nov 17 15:34 ..
-rw-r--r-- 1 root root 19656 Nov  2 17:22 libaep.so
-rw-r--r-- 1 root root 15416 Nov  2 17:22 libatalla.so
-rw-r--r-- 1 root root  6104 Nov  2 17:22 libcapi.so
-rw-r--r-- 1 root root 24104 Nov  2 17:22 libchil.so
-rw-r--r-- 1 root root  6104 Nov  2 17:22 libgmp.so
-rw-r--r-- 1 root root 89112 Nov  2 17:22 libgost.so
-rw-r--r-- 1 root root  6104 Nov  2 17:22 libpadlock.so
-rw-r--r-- 1 root root 23936 Nov  2 17:22 libsureware.so
-rw-r--r-- 1 root root 19624 Nov  2 17:22 libubsec.so
-rw-r--r-- 1 root root 19448 Nov  2 17:22 lib4758cca.so
-rw-r--r-- 1 root root 19704 Nov  2 17:22 libcswift.so
-rw-r--r-- 1 root root 15304 Nov  2 17:22 libnuron.so

[muromec]

Hi,

Due to debian packaging policies, single source package (openssl) is packaged as few separate binary packages, i.e. openssl (command line), libssl1.0.0 (library) and number of debug and documentation packages.

You have probably installed only command line package. Try to install library package like this:

sudo apt-get install libssl1.0.0=1.0.1f-1ubuntu2.7dstu0~trusty

Please take a note, that PPA was not updated in a while and if have intent to use it in production, you'd better rebase patches on latest version.

Hope this helps.

[gmile]

@muromec Thank you Ilya,

Try to install library package like this:

It worked, I can now see the dstu, thanks!

Please take a note, that PPA was not updated in a while and if have intent to use it in production, you'd better rebase patches on latest version.

Yes, this makes total sense, thank you for a reminder.

For now I'm just playing with openssl/dstu combination, trying to verify a signed content with a certificate(s) I have at hand.

[muromec]

For now I'm just playing with openssl/dstu combination, trying to verify a signed content with a certificate(s) I have at had.

Please note, that some files you'd get (especially ones from tax office) would have additional transport headers before ASN1 structure that openssl expects to see, which you need to unpack manually.

[gmile]

@muromec I have a sample file (it contains a small sample string) signed by key obtained from PrivatBank. Can you help me with verifying the contents using PrivatBank cert?

Please note, that some files you'd get (especially ones from tax office) would have additional transport headers before ASN1

The ASN.1 structure seems OK. The output looks like a fine ASN.1 – https://gist.github.com/gmile/a9bb5cb57fc8195d74029251eb3946ba.

But how can I verify the signature using the PrivatBank root certificate? I'm looking at CA-3004751DEF2C78AE010000000100000049000000.cer, downloaded from https://acsk.privatbank.ua/certs?

[gmile]

@muromec in case you would prefer to not discuss this in a github issue (as strictly speaking what I'm asking now is a general openssl question, and is not an issue related to openssl-dstu), I've posted the question here: https://serverfault.com/questions/889799/how-to-verify-signature-on-a-file-using-openssl

[muromec]

There is related package called dstud, also available as static binary on ppa, that is made especially for verifying signatures without much exposure to openssl madness.

You can however use openssl smime subcommand to verify data in command line.

[gmile]

@muromec thank you, I'll take a look at dstud too!

Just tried using smime utility, got this:

openssl smime -verify -in my_message.txt.p7s -engine dstu -inform DER certificate.pem
engine "dstu" set.
Verification failure
140089686927008:error:21075075:PKCS7 routines:PKCS7_verify:certificate verify error:pk7_smime.c:342:Verify error:unable to get local issuer certificate

Does it look like I'm on the right path? I have 4 files with certs from PrivatBank, and only picked the first one (also, converted it from DER to PEM) to try to perform signed message verification:

$ ls -l PrivateCerts/
total 16
-rwxr-xr-x 1 root root 1414 Dec 27 13:27 CA-3004751DEF2C78AE010000000100000049000000.cer
-rwxr-xr-x 1 root root 1721 Dec 27 13:27 CACMPServer-D84EDA1BB9381E8020000000100000019000000.cer
-rwxr-xr-x 1 root root 1590 Dec 27 13:27 CAOCSPServer-D84EDA1BB9381E802000000010000001A000000.cer
-rwxr-xr-x 1 root root 1498 Dec 27 13:27 CATSPServer-3004751DEF2C78AE02000000010000004A000000.cer

[muromec]

You are on a right path for sure. To get it fully working you need to provide openssl with full certificate chain up to CZO root because CAs, such as privatbank don't use self-signed certificates themselves, but ones signed by government agency.

Try verifying certificates themselves first with x509 subcommand.

[gmile]

privatbank don't use self-signed certificates themselves, but ones signed by government agency

@muromec this totally makes sense, thank you.

Why you say I should try to verify the certificate, what does that mean in technical terms? I looked at the issuer, it is indeed CZO:

$ printf "$(openssl x509 -in certificate.pem -issuer -noout)\n"
issuer= /O=Міністерство юстиції України/OU=Адміністратор ІТС ЦЗО/CN=Центральний засвідчувальний орган/serialNumber=UA-00015622-2012/C=UA/L=Київ

Where do I go from there? I suspect I have the following options:

1. download root CZO certificate, and somehow concatenate it with my PrivatBank cert in proper order,
2. download root CZO certificate, put it in a folder along with my PrivatBank cert, tell OpenSSL to use CApath to my folder and hope it will figure things out.

---

Separately, I tried to turn off verification of signature of given certificate (as seen here):

openssl smime -verify -noverify -in my_message.txt.p7s -engine dstu -inform DER
engine "dstu" set.
Hello, world!
Verification failure
139982350653088:error:0D07207B:asn1 encoding routines:ASN1_get_object:header too long:asn1_lib.c:150:
139982350653088:error:0D068066:asn1 encoding routines:ASN1_CHECK_TLEN:bad object header:tasn_dec.c:1306:
139982350653088:error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:831:
139982350653088:error:21071069:PKCS7 routines:PKCS7_signatureVerify:signature failure:pk7_doit.c:1132:
139982350653088:error:21075069:PKCS7 routines:PKCS7_verify:signature failure:pk7_smime.c:410:

I can see my Hello, world! message, but now something seems to be wrong with the ASN.1. Am I hitting the issue you've mentioned, e.g.

Please note, that some files you'd get (especially ones from tax office) would have additional transport headers before ASN1

[muromec]

CZO root is seems to be this one: https://czo.gov.ua/ca-certificates (3004751DEF2C78AE010000000100000001000000).

You should place both privatbank cert and CZO cert in CA.pem and append -CAfile CA.pem to your commandline. You should have "Verify error:unhandled critical extension" after this, as openssl commandline doesn't know about custom extensions present in certificate (this can't be suppresed in commandline).

I just tried all of this myself and it doesn't work for me either (bad ASN1 encoding). My best guess would be that DSTU curve format in openssl code should be changed.

[gmile]

I just tried all of this myself and it doesn't work for me either (bad ASN1 encoding). My best guess would be that DSTU curve format in openssl code should be changed.

@muromec got it. So I suppose there's no way move forward here right now (not event with dstud). I see.

Do you have any plans on supporting the OpenSSL DSTU patch (e.g. fixing the issue, and, maybe rebasing)? Or did you find any other way to move forward with an up-to-date DSTU implementation that is not related to OpenSSL?

I generally don't care much about OpenSSL to be honest. Just want to find something to replace our proprietary solution with something that is open sourced and more or less stable.

[gmile]

Today I've learned about a project called BountyCastle. They claim support of DSTU-4145 scheme https://bouncycastle.org/specifications.html.

I'll explore their examples.

@muromec thank you for your patience and all your help!

[muromec]

After quick debug dive I have found that signature format has changed slightly:

diff --git a/engines/uadstu/dstu_pmeth.c b/engines/uadstu/dstu_pmeth.c
index 8ec0566..e28213b 100644
--- a/engines/uadstu/dstu_pmeth.c
+++ b/engines/uadstu/dstu_pmeth.c
@@ -343,13 +343,12 @@ static int dstu_pkey_verify(EVP_PKEY_CTX *ctx, const unsigned char *sig,
     goto err;
     }

-    field_size = BN_num_bytes(n);
-
-    if (!d2i_ASN1_OCTET_STRING(&dstu_sig, &sig, siglen))
-       goto err;

-    sig = ASN1_STRING_data(dstu_sig);
-    siglen = ASN1_STRING_length(dstu_sig);
+    field_size = BN_num_bytes(n);
+    if (d2i_ASN1_OCTET_STRING(&dstu_sig, &sig, siglen)) {
+        sig = ASN1_STRING_data(dstu_sig);
+        siglen = ASN1_STRING_length(dstu_sig);
+    }

     if (siglen & 0x01)
        goto err;

It was octet string inside octet string before and now it's just raw buffer. With this patch you should have positive verification result on your file.

Try checking out dstu-1_0_1h branch and compiling with this command: ./Configure --prefix=/usr --openssldir=/usr/lib/ssl --libdir=lib/x86_64-linux-gnu no-idea no-mdc2 no-rc5 no-zlib enable-tlsext no-ssl2 enable-uadstu enable-ec_nistp_64_gcc_128 enable-ec_nistp_64_gcc_128 linux-x86_64 && make depend && make

[gmile]

@muromec wow, totally did not expected that. Thank you! Let me try applying patch and see what happens.

Forgive me my ignorance, after running the make line you've provided where will I get a binary file (openssl command?) to run?

[muromec]

binary would be at ./apps/openssl.

To get around unsupported critical extensions, yo can do this:

diff --git a/crypto/pkcs7/pk7_smime.c b/crypto/pkcs7/pk7_smime.c
index a5104f8..bb4eeb3 100644
--- a/crypto/pkcs7/pk7_smime.c
+++ b/crypto/pkcs7/pk7_smime.c
@@ -335,6 +335,7 @@ int PKCS7_verify(PKCS7 *p7, STACK_OF(X509) *certs, X509_STORE *store,
                }
                if (!(flags & PKCS7_NOCRL))
                        X509_STORE_CTX_set0_crls(&cert_ctx, p7->d.sign->crl);
+               cert_ctx.param->flags |= X509_V_FLAG_IGNORE_CRITICAL;
                i = X509_verify_cert(&cert_ctx);
                if (i <= 0) j = X509_STORE_CTX_get_error(&cert_ctx);
                X509_STORE_CTX_cleanup(&cert_ctx);

[gmile]

@muromec after applying both patches verification finally worked, thank you so much!

./apps/openssl smime -verify -noverify -in my_message.txt.p7s -engine dstu -inform DER
engine "dstu" set.
Hello, world!
Verification successful

I tried to verify a message I got from a colleague, with a different error this time (something related to ASN.1 again):

./apps/openssl smime -verify -noverify -in decoded_signed_content.txt -engine dstu -inform DER
engine "dstu" set.
Error reading S/MIME message
139862452258464:error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1319:
139862452258464:error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:381:Type=PKCS7_ISSUER_AND_SERIAL
139862452258464:error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:751:Field=issuer_and_serial, Type=PKCS7_SIGNER_INFO
139862452258464:error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:711:Field=signer_info, Type=PKCS7_SIGNED
139862452258464:error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:751:
139862452258464:error:0D08403A:asn1 encoding routines:ASN1_TEMPLATE_EX_D2I:nested asn1 error:tasn_dec.c:579:Field=d.sign, Type=PKCS7

I'm going to ask my colleague how exactly did he apply the signature (I signed a file by providing a file and a private key at https://czo.gov.ua/sign), as I was sure there should be no difference with my hello_world file.

---

On a separate note, how safe by your own standards it would be to keep this line in production?

cert_ctx.param->flags |= X509_V_FLAG_IGNORE_CRITICAL;

[muromec]

I believe it's better to figure out which one it is and add it to crypto/x509v3/v3_purp.c so it would stop being unhandled.

[gmile]

@muromec I've converted the certs to PEM and concatenated them:

openssl x509 -inform der -in CZOROOT.cer -engine dstu > czo.pem
openssl x509 -inform der -in CA-3004751DEF2C78AE010000000100000049000000.cer -engine dstu > privat.pem
cat czo.pem privat.pem > bundle.pem

By the way, it seems like the concat order doesn't matter here? I'm not sure what will happen if I have a chain of 3 certs though.

This is the first time it did this, and it worked:

./apps/openssl smime -verify -CAfile bundle.pem -in my_message.txt.p7s -engine dstu -inform DER
engine "dstu" set.
Hello, world!
Verification successful

I'm posting it here in case someone passes by.

[muromec]

Actually the unhandled extension seems to be NID_qcStatements which is important, as it denotes that certificate is qualified to produce legally recognizable signatures. Not only should it be handled, but also all signatures without this extension should be rejected. So instead of ignoring it as done in previous patch, add it to white list like this:

diff --git a/crypto/x509v3/v3_purp.c b/crypto/x509v3/v3_purp.c
index f59bfc1..7e564dc 100644
--- a/crypto/x509v3/v3_purp.c
+++ b/crypto/x509v3/v3_purp.c
@@ -292,6 +292,7 @@ int X509_supported_extension(X509_EXTENSION *ex)
        NID_basic_constraints,  /* 87 */
        NID_certificate_policies, /* 89 */
            NID_ext_key_usage,  /* 126 */
+       NID_qcStatements, /* 286 */
 #ifndef OPENSSL_NO_RFC3779
        NID_sbgp_ipAddrBlock,   /* 290 */
        NID_sbgp_autonomousSysNum, /* 291 */

This does only half of job however, as you should somehow check for it's existence and value.

[muromec]

By the way, you can produce signatures with online tools, like https://czo.gov.ua/sign and my own, built in JS: https://dstucrypt.github.io/signerbox2 (sources available).

[gmile]

By the way, you can produce signatures with online tools, like https://czo.gov.ua/sign and my own, built in JS: https://dstucrypt.github.io/signerbox2 (sources available).

Thanks, I was using https://czo.gov.ua/sign already, but it's good to know there's now an alternative!

What I'm looking forward to though is automating signing files from OpenSSL+DSTU now. I haven't done any research in that direction yet. This should help me tremendously with testing the system I'm working on (we have a business process, where a file sign/verification happens).

[gmile]

@muromec potentially the files I have may be signed by a private keys, issued by different certificate authorities. How would I go about verification in this case, when I need to keep certificates from all authorities from this list https://czo.gov.ua/ca-registry? E.g. not only from PrivatBank.

Is it possible to group a bunch of certificate (or certificate chains) together in some form (under a directory, or as a sort of package?), before passing them to openssl smime?

I'm thinking of something like:

1. download all certificates from https://czo.gov.ua/ca-registry
2. put them in the same folder
3. specify -CApath paramter to openssl smime,
4. expect it to look and pick a valid certificate for signature verification.

I just tried this by copying bundle.pem I made earlier into /openssl-sandbox/trusted/. This didn't work as I expected:

./apps/openssl smime -verify -CApath /openssl-sandbox/trusted/ -in /yo/my_message.txt.p7s -engine dstu -inform DER
engine "dstu" set.
Verification failure
140042814707360:error:21075075:PKCS7 routines:PKCS7_verify:certificate verify error:pk7_smime.c:342:Verify error:unable to get local issuer certificate

[gmile]

So instead of ignoring it as done in previous patch, add it to white list like this:

@muromec by the way, tried this patch. It works perfectly. I'm yet to internalize what you said there about checking the existence and value of NID_qcStatements. Maybe you can share some resource where I can learn more about it? I'm trying to understand the significance of it.

[muromec]

Maybe you can share some resource where I can learn more about it? I'm trying to understand the significance of it.

This thing:

[muromec]

Is it possible to group a bunch of certificate (or certificate chains) together in some form (under a directory, or as a sort of package?), before passing them to openssl smime?

Sure, you can do it like this: https://github.com/dstucrypt/dstu-validator/tree/master/CA.

Proper filename can be formed with following command openssl x509 -in something-someting.cer -inform DER -noout -subject_hash (and append .0).

[gmile]

This thing...

@muromec much appreciated!

Proper filename can be formed with following command

Worked like a charm, thanks!

[gmile]

@muromec I'm currently puzzled as to how to use .jks file, provided by PrivatBank, to sign anything. I've posted a question here – https://stackoverflow.com/questions/48014312. I'm interested in being able to sign files from the command line (should be very useful for scripting / writing tests).

I think it may be since keytool does not recognize dstu-4145 scheme. But maybe it's something else.

I can see that https://dstucrypt.github.io/signerbox2 works with .jks files perfectly fine. I got increasingly interested in this code now:

https://github.com/dstucrypt/jksreader/blob/24671cc2d6651290eaecd9cd379a8f53486b8ac9/parse.js#L30-L45

It looks cool! I'm going to try and figure out how to write a simple one-time node script and use functions from decode.js and parse.js.

Thanks a ton for all the work you're doing on DSTU-4145 support!

[gmile]

@muromec I've written a small program in node.js in order to extract the key (presumably in .pkcs12 format?):

(update the code below to also extract certificates)

const fs = require('fs');
      jksreader = require('jksreader'),

      pathToFile = process.argv[2],
      password = process.argv[3],

      contents = fs.readFileSync(pathToFile),
      parsedContent = jksreader.parse(contents);

var   key = jksreader.decode(parsedContent.material[0].key, password);

fs.writeFileSync('key', key);

for (var i = 0; i < parsedContent.material[0].certs.length; i++) {
  var cert = parsedContent.material[0].certs[i];
  fs.writeFileSync('cert' + i, cert);
}

I can now parse the key contents via asn1parse:

openssl asn1parse -in key -inform DER

Here's the output (I've erased the actual HEX values with X) – https://gist.github.com/gmile/226bcab15b4317ece4c99d4ec26c47d8.

Where do I go from here? Should I also extract the cert file (the one called after my name)?

The reason I'm doing all this is to be able to sign files using patched openssl from on my computer.

[muromec]

Where do I go from here? Should I also extract the cert file (the one called after my name)?

That's pretty much it. You can try and sign files with openssl smime -sign, provide key, matching certificate (beware, there are two keys and two certificates). I don't remember if it ever worked, but it probably should.

[gmile]

After I extracted cert0, cert1, cert2, cert3 and key, I converted them all from DER to PEM.

I tried all 4 certs with key combinations. 3 of the certs returned "Key mismatch" error:

140337353254560:error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch:x509_cmp.c:330:
140337353254560:error:2108907F:PKCS7 routines:PKCS7_sign_add_signer:private key does not match certificate:pk7_smime.c:170:

One cert returned "Segmentation fault" error, unfortunately:

$ /root/openssl-dstu/apps/openssl smime \
  -sign \
  -in my_message.txt \
  -signer cert1.pem \
  -text \
  -out my_message.txt.signed \
  -inkey key.pem
Segmentation fault

I suppose cert1.pem + key is the right combination, but it fails to work.

Looks like a dead-end to me :-(

[muromec]

Just checked, it seems to work on my setup and produce signature file, but it's rejected by czo online verification tool anyway.

Segfault seems unusual, but that should be relatively easy to debug with GDS, I believe.

[muromec]

Btw, you shouldn't have to guess which certificate is right, since you find correct one by looking into output of ./apps/openssl x509 -in cert1.pem -noout -text -nameopt oneline,utf8,-esc_msb command.

Correct one should have your name and this text: X509v3 Key Usage: critical Digital Signature, Non Repudiation

[gmile]

./apps/openssl x509 -in cert1.pem -noout -text -nameopt oneline,utf8,-esc_msb

@muromec I could not figure out how to correctly display escaped sequences as characters. So this is useful, thank you!

Segfault seems unusual, but that should be relatively easy to debug with GDS, I believe.

Debugging a segfault will take me days :-( I'm considering looking towards BountyCastle. I'm neither C nor Java programmer, so each approach (debug C code or put together Java application from scratch) will likely to consumer a considerable amount of time.

Meanwhile I published a docker image (github, dockerhub) that illustrates the Segmentation Fault issue.

I got all your suggestions in. It also contains a small CLI app (github, npm) to unpack .jks files, based on your work with jksreader.

When starting container, I pass in the key I obtained from PrivatBank. So it all becomes a one-liner:

$ docker run --rm --mount type=bind,source=/Users/gmile/Dropbox/pb_322851xxxx.jks,target=/pb_key.jks -it gmile/openssl-dstu-issue sh end-to-end.sh my_key_password
Extracting key and certs from pb_key.jks...
Written file: /tmp/workdir/key.der
Written file: /tmp/workdir/cert0.der
Written file: /tmp/workdir/cert1.der
Written file: /tmp/workdir/cert2.der
Written file: /tmp/workdir/cert3.der
Converting extracted key from DER to PEM...
engine "dstu" set.
Looking for matching certificate...
Found matching cert is: /tmp/workdir/cert1.der
Converting /tmp/workdir/cert1.der from DER to PEM...
Creating a sample message...
Attempting to sign the message...
Segmentation fault
Cleaning up the mess...

In case you ever come back to this issue, maybe you could make use of the reproducible build I made to get to the bottom of the segmentation fault.

[muromec]

Debugging a segfault will take me days :-(

Not really. Just running debug build under gdb would show you which place in code produces the fault. I believe it can be key format specific, since I used master key certificates myself, not privatbank ones.

Regarding BC, here is complete example of signing file in BC (C# version) that produces signed files accepted by czo verificator: https://gist.github.com/muromec/cfe8da5135084bea50aff5670bde110f

[muromec]

Can't reproduce segfault with pb keys either. Can you please pinpoint location in code where it fails?

[gmile]

Can't reproduce segfault with pb keys either.

@muromec to double check: are you using the docker command I posted above? Just want to confirm our environments match.

If you have docker runtime installed, running the command will do everything for you, e.g. download image, run container, run end-to-end test and finally delete the container.

The only thing you have to do before running the command is to specify the location to the key from PrivatBank on your computer:

$ docker run --rm --mount type=bind,source=/path/to/privatbank/key.jks,target=/pb_key.jks -it gmile/openssl-dstu-issue sh end-to-end.sh your_key_password

This is how image looks like – https://github.com/gmile/openssl-dstu-issue/blob/master/Dockerfile. It pretty much contains your instructions per which branch to use, what patches to apply and how to build openssl. Here's the actual end-to-end test to replicate segmentation fault – https://github.com/gmile/openssl-dstu-issue/blob/master/end-to-end.sh.

---

If you are using the docker command and still don't get the segmentation fault then, I'm confused. Could the privatbank keys we're using be different somehow? I've generated mine recently, like around 2 weeks ago.

Can you please pinpoint location in code where it fails?

I will need some guidance to do this.

Just running debug build under gdb would show you which place in code produces the fault.

How do I make a debug build of openssl? I suspect it must be some config option for ./Configure?

[muromec]

Well, it's really simple.

# do some cleanup first
git clean -fxd
# configure as usual, but prepending debug to target
./Configure --prefix=/usr --openssldir=/usr/lib/ssl --libdir=lib/x86_64-linux-gnu no-idea no-mdc2 no-rc5 no-zlib enable-uadstu enable-ec_nistp_64_gcc_128 enable-ec_nistp_64_gcc_128  debug-linux-x86_64
# build it
make depend && make -j 8
# run debug-build under gdb
gdb ./apps/openssl
# at gdb prompt
run smime \
  -sign \
  -in my_message.txt \
  -signer cert1.pem \
  -text \
  -out my_message.txt.signed \
  -inkey key.pem

It would stop at segfault and would show you backtrace (bt command in gdb). Here is command reference for gdb: http://visualgdb.com/gdbreference/commands/backtrace

[muromec]

My best guess right now is the bug in asn1 parsing code, maybe ever that part we patched right before.

[gmile]

@muromec thanks for the debug build instructions! I finally got back at it.

Here's output from gdb's run:

(gdb) run
Starting program: /openssl-sandbox/openssl-dstu/apps/openssl smime -sign -in /tmp/workdir/message.txt -out /tmp/workdir/message.txt.signed -signer /tmp/workdir/signer.pem -inkey /tmp/workdir/key.pem -text

Program received signal SIGSEGV, Segmentation fault.
0x00000000005b167d in dstu_md_ctrl (ctx=0x0, cmd=2, p1=0, p2=0x7fffffffddd0) at dstu_md.c:83
83          struct dstu_digest_ctx *c = ctx->md_data;

Here's output from gdb's bt:

(gdb) bt
#0  0x00000000005b167d in dstu_md_ctrl (ctx=0x0, cmd=2, p1=0, p2=0x7fffffffddd0) at dstu_md.c:83
#1  0x00000000005ed53e in asn1_write_micalg (out=0x956d30, mdalgs=0x9572c0) at asn_mime.c:219
#2  0x00000000005ed805 in SMIME_write_ASN1 (bio=0x956d30, val=0x956ad0, data=0x956c60, flags=21569, ctype_nid=22, econt_nid=0, mdalgs=0x9572c0, it=0x65e000 <PKCS7_it>) at asn_mime.c:311
#3  0x0000000000584352 in SMIME_write_PKCS7 (bio=0x956d30, p7=0x956ad0, data=0x956c60, flags=21569) at pk7_mime.c:89
#4  0x000000000044b9ec in smime_main (argc=11, argv=0x7fffffffe6c0) at smime.c:788
#5  0x0000000000403196 in do_cmd (prog=0x9422e0, argc=11, argv=0x7fffffffe6c0) at openssl.c:490
#6  0x0000000000402da9 in main (Argc=11, Argv=0x7fffffffe6c0) at openssl.c:382

Does that give any clue at what the problem might be here?

[gmile]

Btw I saw that running gdb in docker produces the following error:

warning: Error disabling address space randomization: Operation not permitted

I then learned that I have to start the container with a few additional flags. Here's the final line to start container:

docker run --rm --mount type=bind,source= /path/to/privatbank/key.jks,target=/pb_key.jks -it --cap-add=SYS_PTRACE --security-opt seccomp=unconfined gmile/openssl-dstu-issue sh end-to-end.sh your_key_password

[muromec]

Does that give any clue at what the problem might be here?

Yep, that looks like access to 0x0 address, which can be permitted but obviously is a bug.

[muromec]

try this:

 dstu_ameth.o: ../../include/openssl/cms.h ../../include/openssl/crypto.h
diff --git a/engines/uadstu/dstu_md.c b/engines/uadstu/dstu_md.c
index f69f9bf..74d09ea 100644
--- a/engines/uadstu/dstu_md.c
+++ b/engines/uadstu/dstu_md.c
@@ -79,6 +79,9 @@ static int dstu_md_cleanup(EVP_MD_CTX *ctx)

 static int dstu_md_ctrl(EVP_MD_CTX *ctx, int cmd, int p1, void *p2)
     {
+    if (!ctx) {
+        return 0;
+    }
     gost_subst_block sbox;
     struct dstu_digest_ctx *c = ctx->md_data;

[gmile]

@muromec this works, thank you!

[gmile]

Next steps I'm going to take is to try to figure out how to do TSP / OCSP verification.

Thanks for all your help!

[gmile]

@muromec do you know where the dstu engine .so file is located?

As usually, after applying all patches and running make, this time I ran make install and ended up with system-wide openssl that seems to be dstu-aware:

# openssl engine -t dstu
(dstu) Reference implementation of DSTU engine
     [ available ]

However I can't reference the dstu engine from another language (in my case – from erlang; it links to system openssl dynamically):

# erl
Erlang/OTP 20 [erts-9.2] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:10] [hipe] [kernel-poll:false]

Eshell V9.2  (abort with ^G)
1> {ok, Engine} = crypto:engine_load(<<"dstu">>, [], []).
** exception error: no match of right hand side value {error,bad_engine_id}

At the same time gost engine can be referenced with no problem:

# erl
Erlang/OTP 20 [erts-9.2] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:10] [hipe] [kernel-poll:false]

Eshell V9.2  (abort with ^G)
1> {ok, Engine} = crypto:engine_load(<<"gost">>, [], []).
{ok,#Ref<0.950608548.1984823298.71941>}

I learned that custom engines should be referenced differently, by specifying dynamic path to the engine:

{ok, Engine} = crypto:engine_load(<<"dynamic">>,
                                      [{<<"SO_PATH">>,
                                        <<"/some/path/otp_test_engine.so">>},
                                       {<<"ID">>, <<"MD5">>},
                                       <<"LOAD">>],
                                      []).

So I think I need to know where the dstu .so file is, but I can't seem to find in the system it either, by running variants of find:

# find / -name dstu*
# find / -name *dstu

[muromec]

It probably would not work anyway if openssl itself was not patched. You better try to repackage OpenSSL_1_0_2-dstu branch from this repo and install it in system, adjust LD_LIBRARY_PATH so erlang would use dstu version from other location.

Engines were meant to be used to offload already known cypher to hw implementation, not to add new ones. They are not plugins and can't add cyphers that openssl binary had no idea at compilation time.

[muromec]

As to why you can't find dynamic library, it's probably just included in your openssl library.

[gmile]

It probably would not work anyway if openssl itself was not patched. You better try to repackage OpenSSL_1_0_2-dstu branch from this repo and install it in system

Initially I used dstu-1_0_1h branch with all 3 patches you provided in comments https://github.com/dstucrypt/openssl-dstu/issues/2#issuecomment-354297104, https://github.com/dstucrypt/openssl-dstu/issues/2#issuecomment-354308772 and https://github.com/dstucrypt/openssl-dstu/issues/2#issuecomment-357723329. After applying patches I ran:

./Configure --prefix=/usr \
            --openssldir=/usr/lib/ssl \
            --libdir=lib/x86_64-linux-gnu \
            no-idea \
            no-mdc2 \
            no-rc5 \
            no-zlib \
            enable-uadstu \
            enable-ec_nistp_64_gcc_128 \
            debug-linux-x86_64 && \
make depend && \
make && \
make install

So this installed openssl with dstu system-wide:

# openssl engine -t dstu
(dstu) Reference implementation of DSTU engine
     [ available ]

Only after I had openssl with dstu, I moved on to installing erlang package (from .deb file), which I believe links to whatever openssl is in the system (meaning it probably reads default LD_LIBRARY_PATH; btw I don't know anything about LD_LIBRARY_PATH and how).

Since there was no prior openssl in the system, and since erlang .deb package was not compiled with openssl linked statically.... this all suggests erlang should have picked up my very own built openssl with dstu support.

---

If you're saying I should try OpenSSL_1_0_2-dstu branch, I will do that (btw it lacks the final patch you've proposed, e.g. this https://github.com/dstucrypt/openssl-dstu/issues/2#issuecomment-357723329 – is it OK?).

[gmile]

Btw, noted on the actual engines purpose and the fact that openssl only knows about things that it knew about during compile time, thank you.