[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)

[OdedRaiches]

I'm trying to use this library for communication with Equinix SmartKey via KMIP interface (which they say is supported). I have this "certificate verify failed" exception but not sure what's failing on my end. I did a self signed certificate and also tried to download the server's certificate for the "ca_certs". This is my configuration:

[client]
host=eu.smartkey.io
port=5696
certfile=/mnt/homes/oded/tmpProj/pykmip/example-com.cert.pem
keyfile=/mnt/homes/oded/tmpProj/pykmip/example-com.key.pem
ca_certs=/mnt/homes/oded/tmpProj/pykmip/smartkeycert.cer
cert_reqs=CERT_REQUIRED
ssl_version=PROTOCOL_TLS
do_handshake_on_connect=True
suppress_ragged_eofs=True
username=*** (email of smartkey user)
password=***  (password of  smartkey user)

I run the example in kmip/demos/pie/get.py and get this:

Traceback (most recent call last):
  File "get.py", line 48, in <module>
    config_file=config_file
  File "/usr/local/lib/python3.6/dist-packages/kmip/pie/client.py", line 1745, in __enter__
    self.open()
  File "/usr/local/lib/python3.6/dist-packages/kmip/pie/client.py", line 173, in open
    self.proxy.open()
  File "/usr/local/lib/python3.6/dist-packages/kmip/services/kmip_client.py", line 275, in open
    self.socket.connect((self.host, self.port))
  File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
    raise value
  File "/usr/local/lib/python3.6/dist-packages/kmip/services/kmip_client.py", line 275, in open
    self.socket.connect((self.host, self.port))
  File "/usr/lib/python3.6/ssl.py", line 1109, in connect
    self._real_connect(addr, False)
  File "/usr/lib/python3.6/ssl.py", line 1100, in _real_connect
    self.do_handshake()
  File "/usr/lib/python3.6/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)

[PeterHamilton]

Hi @OdedRaiches, thanks for filing this issue. Going from the description of your setup, your self-signed certificate is probably the problem here. You likely need to get a client certificate that is signed and trusted by your server's CA. Your server doesn't recognize your self-signed certificate, causing certificate verification to fail.

[OdedRaiches]

@PeterHamilton thanks for the quick reply! I found that ca_certs is needed even though Equinix is signed by a known CA (Digicert). I passed this part but got another error while trying the kmip/demos/pie/get.py example script: demo - ERROR - OPERATION_FAILED: PERMISSION_DENIED - UUID parse error: Invalid length; expecting 32 or 36 chars, found 26

As a uid I pass this: '6951d846-62ee-4702-a004-ed839ea679ce' Which is 36 chars of course. This is The UUID I got after creating a key in the SmartKey dashboard:

Am I missing something? When I look at the 'usage' documentation I see that the key_id is built somehow but I don't understand how its got to do with the UUID that the object has (and why is it different from the demo anyway?).

[PeterHamilton]

That is odd. The UUID should be what you need to fetch the key using get.py. That error is coming back from the SmartKey server, which implies that the UUID is getting truncated by PyKMIP before it gets sent to the server. But that shouldn't ever happen...

Tweak get.py and change L25:

logger = utils.build_console_logger(logging.INFO)

to

logger = utils.build_console_logger(logging.DEBUG)

Send me the updated log output; from that I'll be able to tell if PyKMIP is sending the full UUID to the SmartKey server. Note that this will include the unencrypted encoded content for your SmartKey username and password, so if you can't share that I'll have to walk you through what specific piece of the log output I need. You can send me the log output over email if you prefer to avoid a public channel like this.

[OdedRaiches]

@PeterHamilton this is the only message I get back (same as before): 2020-12-02 17:17:03,311 - demo - ERROR - OPERATION_FAILED: PERMISSION_DENIED - UUID parse error: Invalid length; expecting 32 or 36 chars, found 26

I also used pdb and watched what is the content of stream.buffer when the code reaches _send_message and I see my UUID in plaintext as part of it, so I guess its transferred alright.

[PeterHamilton]

Hmmm, setting logging to debug should definitely have dumped that stream.buffer to stdout. That buffer should contain a long hex string which is the encoded KMIP message for the SmartKey server. Look for the following hex bytes:

420094

They should be towards the end of the buffer. Anything after those bytes should be the encoded UUID (plus some leading metadata). Send me that.

[OdedRaiches]

@PeterHamilton Yeah that's weird. Anyway, that's what pdb shows:

1724        def _send_message(self, message):
1725            stream = BytearrayStream()
1726            message.write(stream, self.kmip_version)
1727 ->         self.protocol.write(stream.buffer)
1728 
(Pdb) stream.__dict__
{'buffer': b'B\x00x\x01\x00\x00\x00\xf8B\x00w\x01\x00\x00\x00\xa0B\x00i\x01\x00\x00\x00 B\x00j\x02\x00\x00\x00\x04\x00\x00\x00\x01\x00\x00\x00\x00B\x00k\x02\x00\x00\x00\x04\x00\x00\x00\x02\x00\x00\x00\x00B\x00\x0c\x01\x00\x00\x00`B\x00#\x01\x00\x00\x00XB\x00$\x05\x00\x00\x00\x04\x00\x00\x00\x01\x00\x00\x00\x00B\x00%\x01\x00\x00\x00@B\x00\x99\x07\x00\x00\x00\x1a**************************\x00\x00\x00\x00\x00\x00B\x00\xa1\x07\x00\x00\x00\x0c************\x00\x00\x00\x00B\x00\r\x02\x00\x00\x00\x04\x00\x00\x00\x01\x00\x00\x00\x00B\x00\x0f\x01\x00\x00\x00HB\x00\\\x05\x00\x00\x00\x04\x00\x00\x00\n\x00\x00\x00\x00B\x00y\x01\x00\x00\x000B\x00\x94\x07\x00\x00\x00$6951d846-62ee-4702-a004-ed839ea679ce\x00\x00\x00\x00'}

I put a bunch of *** instead of my username and password (username first and password second). I made them of the same length as they were. I kept the UUID in the clear, its just for testing.

[PeterHamilton]

Ok, so the relevant part is this segment, at the end:

B\x00\x94\x07\x00\x00\x00$6951d846-62ee-4702-a004-ed839ea679ce\x00\x00\x00\x00

The \x00$ specifically refers to the length of the UUID. In this case, $ is hex 24, or 36 bytes (this ignores the four \x00 padding bytes at the end). So that's correct.

I do not know why the SmartKey server is only seeing 26 characters since PyKMIP is sending all 36. Is there a way to enable additional logging on the SmartKey server so we can see what it is receiving from the client?

[OdedRaiches]

@PeterHamilton no way to see any additional logs on server side, only the dashboard with very limited logging. It might concern a VM that I'm using. I'll try doing a test on another environment and see if it somehow matters. Thanks!

[OdedRaiches]

@PeterHamilton Tried on another environment and got the same error. Can you maybe try to debug this SmartKey KMIP interface? Registering to the free trial takes a few minutes (no need for credit card or anything): https://www.equinix.com/services/edge-services/smartkey/

[OdedRaiches]

@PeterHamilton Hi, did another test using libkmip. I ran the demo_get.c code with the same credentials and certificates as in the Pykmip example. The code returns this:

The KMIP operation was executed with no errors.
Result: Operation Failed (1)

I did some debugging with gdb and saw this:

printing : resp_item.result_status -> KMIP_STATUS_OPERATION_FAILED
printing : resp_item.result_reason -> KMIP_REASON_GENERAL_FAILURE
printing : resp_item->result_message.value -> "expected BatchCount, got TimeStamp"

Now the server is not complaining about the UUID but something else. I guess pykmip and libkmip should write similar payload so not sure if theirs any sense to this. Is it ok that I didn't file this issue under the libkmip? I see there is already a similar issue in the libkmip repo (also, I'm not sure anymore if this is a server or a client problem).

[PeterHamilton]

So that is an odd error. The TimeStamp field is an optional field in the RequestHeader which is found right before the BatchCount field. The fact that the SmartKey server is complaining about it makes me think that the server doesn't support all of the optional fields that are defined by the KMIP specification.

To test this, you'll have to modify some libkmip internals. You can either set the TimeStamp field to zero here or you can just disable TimeStamp field encoding entirely, by commenting out the if-block here. Run make clean and then make again to rebuild demo_get.c and retest. I'm curious if the server likes that modified request better.

Finally, what KMIP version is the SmartKey server using? The libkmip demo_get.c uses KMIP v1.0, while the PyKMIP pie/get.py demo script will use KMIP v1.2. Generally speaking this isn't a problem; the KMIP v1.* series is backwards compatible. However, if the SmartKey server is using a KMIP v2.* version, then that may explain the problem. KMIP v2.* breaks backwards compatibility with KMIP v1.*. Both PyKMIP and libkmip support KMIP v2.0, so if the server is using KMIP v2.0 we can pivot you to test with that. There is a new KMIP v2.1 specification out (or coming out soon) which neither library yet supports.

[OdedRaiches]

@PeterHamilton I set the TimeStamp field with zero and got the UUID parse error again:

(gdb) p result
$1 = KMIP_STATUS_OPERATION_FAILED
(gdb) p resp_item.result_reason
$2 = KMIP_REASON_PERMISSION_DENIED
(gdb) p resp_item->result_message.value
$3 = 0x5555557c7f90 "UUID parse error: Invalid length; expecting 32 or 36 chars, found 26"

So that looks more like before. About the KMIP version that SmartKey uses - I don't know (but sounds like that may be the problem). How can we test this with KMIP v2.0?

[OdedRaiches]

@PeterHamilton tried running pykmip with KMIP_2_0 and got this error: 2020-12-03 16:52:18,329 - demo - ERROR - OPERATION_FAILED: FEATURE_NOT_SUPPORTED - unsupported protocol version

So I guess that SmartKey does not support 2.0 (?)

[OdedRaiches]

@PeterHamilton I saw that SmartKey is "powered by Fortanix" and on Fortanix site I found this: "Our KMIP Server supports version 1.0, 1,1, 1,2, 1.3, and 1.4." https://support.fortanix.com/hc/en-us/articles/360050835232-Which-KMIP-Server-versions-do-we-support-

I tried on all the supported version and got the same UUID parse error as before.

[PeterHamilton]

@OdedRaiches If the SmartKey server is returning the same error for both PyKMIP and libkmip, which are different client implementations, my gut says there's something weird with how the server is parsing the client request. Maybe it's expecting another optional field that we're not sending?

Try creating a key using PyKMIP or libkmip. You should get back the UUID for your new key. Then try retrieving that key using that UUID. Maybe the UUID displayed in the web interface is not the UUID you should be using to fetch it?

I won't have a ton of time today to debut this but I'm still interested to hear how the above test goes.

[OdedRaiches]

@PeterHamilton I still get the same UUID parse error even though I didn't give the UUID as a parameter to the create request. Here is the buffer again:

(Pdb) stream.__dict__
{'buffer': b'B\x00x\x01\x00\x00\x01\x88B\x00w\x01\x00\x00\x00\xa0B\x00i\x01\x00\x00\x00 B\x00j\x02\x00\x00\x00\x04\x00\x00\x00\x01\x00\x00\x00\x00B\x00k\x02\x00\x00\x00\x04\x00\x00\x00\x02\x00\x00\x00\x00B\x00\x0c\x01\x00\x00\x00`B\x00#\x01\x00\x00\x00XB\x00$\x05\x00\x00\x00\x04\x00\x00\x00\x01\x00\x00\x00\x00B\x00%\x01\x00\x00\x00@B\x00\x99\x07\x00\x00\x00\x1a**************************\x00\x00\x00\x00\x00\x00B\x00\xa1\x07\x00\x00\x00\x0c************\x00\x00\x00\x00B\x00\r\x02\x00\x00\x00\x04\x00\x00\x00\x01\x00\x00\x00\x00B\x00\x0f\x01\x00\x00\x00\xd8B\x00\\\x05\x00\x00\x00\x04\x00\x00\x00\x01\x00\x00\x00\x00B\x00y\x01\x00\x00\x00\xc0B\x00W\x05\x00\x00\x00\x04\x00\x00\x00\x02\x00\x00\x00\x00B\x00\x91\x01\x00\x00\x00\xa8B\x00\x08\x01\x00\x00\x000B\x00\n\x07\x00\x00\x00\x17Cryptographic Algorithm\x00B\x00\x0b\x05\x00\x00\x00\x04\x00\x00\x00\x03\x00\x00\x00\x00B\x00\x08\x01\x00\x00\x000B\x00\n\x07\x00\x00\x00\x14Cryptographic Length\x00\x00\x00\x00B\x00\x0b\x02\x00\x00\x00\x04\x00\x00\x01\x00\x00\x00\x00\x00B\x00\x08\x01\x00\x00\x000B\x00\n\x07\x00\x00\x00\x18Cryptographic Usage MaskB\x00\x0b\x02\x00\x00\x00\x04\x00\x00\x00\x0c\x00\x00\x00\x00'}
(Pdb) c
2020-12-08 09:43:26,027 - demo - ERROR - OPERATION_FAILED: PERMISSION_DENIED - UUID parse error: Invalid length; expecting 32 or 36 chars, found 26

I put *** instead of my username and password in the same length.

[PeterHamilton]

You definitely should not be getting that error for a Create request. The server is assigning your new key a UUID and sending it back to you. If the server is parsing a UUID, it is doing something wrong.

I'm going to pick through the above buffer dump to make doubly sure we're not sending anything weird to the server.

[PeterHamilton]

I decoded the buffer manually to make sure there weren't any unexpected contents. I couldn't find any. This is the message that corresponds to your buffer:

RequestMessage
  RequestHeader
    ProtocolVersion
      ProtocolVersionMajor, Value: 1
      ProtocolVersionMinor, Value: 2
    Authentication
      Credential
        CredentialType, Value: Username and Password
        CredentialValue
          Username, Value: **************************
          Password, Value: ************
    BatchCount, Value: 1
  BatchItem
    Operation, Value: Create
    RequestPayload
      ObjectType, Value: SymmetricKey
      TemplateAttribute
        Attribute
          AttributeName, Value: Cryptographic Algorithm
          AttributeValue, Value: AES
        Attribute
          AttributeName, Value: Cryptographic Length
          AttributeValue, Value: 256 bits
        Attribute
          AttributeName, Value: Cryptographic Usage Mask
          AttributeValue, Value: ENCRYPT | DECRYPT

This is a valid Create request. If the SmartKey server doesn't accept this, then it's either expecting an optional field that we're not sending or it's failing to parse the message properly. There's not much else I can do debug wise here. This message encoding is almost identical to some of the encodings I test against which are published in the KMIP specification testing document.

One final theory - is your user allowed to create and retrieve keys? Is it possible that the Permission Error is sending back a bogus error message that's actually caused by your user account permissions?

[PeterHamilton]

One final final thought - the only thing that is 26 characters in length in the request is your username. Is there a UUID for your user account that you should be using as your username instead? If there is, try that.

[OdedRaiches]

@PeterHamilton Ok, now it works. Some of my findings with Equinix Smartkey if anyone else has trouble: When creating an app:

1. If the credential type is API key: a. For certificates: only ca_certs parameter is required for authentication (and certfile and keyfile are optional) b. For username/password: you take them from the credentials in the app like so:

2. If the credential type is certificate: a. For certificates: you need ca_certs, certfile and keyfile. b. For username/password: you take the UUID of the app itself as a username and the password is blank.

For testing purposes - the certificate can be self-signed, no need for it to be signed by a known CA. The nice thing in Equinix SmartKey is that you can see what app created the Security object and from there you know how to access this security object afterwards. IMO, its better using the API key credentials since they are also available from the console itself in comparison to certificates which only holds the public part that was uploaded (and private part should be kept safe at the client side).

Thanks for all the help!

[PeterHamilton]

@OdedRaiches No problem! Really glad to hear that you figured it out. After my debug yesterday I wondered if the credentials were the issue. I'm also happy to hear that SmartKey works correctly.

If you run into any future problems with either PyKMIP or libkmip, don't hesitate to let me know. Cheers!