Add generic CustomAttribute support

[PeterHamilton]

Per #459, the PyKMIP server will fail to decode custom attributes that are not encoded as text strings. Updating custom attribute support, including encoding/decoding, to support any KMIP primitive type would allow the PyKMIP server to better fit custom use cases normally reserved for specific server and vendor types.

This feature could include various side features as well, including:

• improved logging to note when custom attributes are found by the server
• attribute configuration options allowing users to define custom attributes for the server

The library code that specifically broke in #459 can be found here and here.

[schlutech]

The TTLV format lends itself to automatically detecting the encoding (type) from the 4th byte per tag. This could eliminate the need for users to configure for custom attributes.

In a similar way, custom attributes, to be useful, would need to be added to the database for register/create operations.

Lastly, to be able via the "locate" operation to filter via Attributes other than "Name" (such as custom attributes) could have far reaching implications as well; above and beyond custom attributes.

[PeterHamilton]

Agreed, this would definitely work for basic primitive attributes (e.g., text strings, integers, booleans). It would be trickier for complex attributes (i.e., structures) but the simplest approach would be to punt on decoding the structure and instead just store the encoding of it and do a simple comparison on attribute encodings. To be honest, that may be the simplest way to support custom attributes in general, especially when dealing with storing them in the database; adding custom attribute tables for different attribute types and structures is just not feasible without sophisticated automation.

Upgrading how Locate filters based on arbitrary attributes has also been a long-term wanted feature for me but it just never gets priority. I'll create a separate issue for that as well.

[schlutech]

Its hard for me to think of how a client could use complex custom attributes in a meaningful way. If that kind of complexity was required I would assume the client application would choose to store the information in multiple "key/value" (attribute/value) pairs or byte-encode the data of base64 it into a text string, etc.

Just being able to take Custom Attributes and store them as "meta-data" with the managed object seems ideal.

[PeterHamilton]

True. I'm thinking here of just broadly supporting the KMIP spec as much as possible. A client would be free to register custom attributes however it wants. How the server stores those and does the internal filtering/comparison should be opaque to the client but should support simple and complex custom attributes.

[cocomarin]

I don´t know if my issue is related to this or I have to open another thread, but I´ll start here: We are trying to manage an IBM TS4300 Library with PyKMIP, and we are having issues with so attributes the client is sending. This is a snap of the server.log, after the SSL Handshake is successful: 2019-06-11 12:24:37,956 - kmip.server.session.00000002 - DEBUG - Session cipher selected: ('AES256-SHA256', 'TLSv1.2', 256) 2019-06-11 12:24:37,957 - kmip.server.session.00000002 - INFO - Session client identity: IBMLibrary 2019-06-11 12:24:37,959 - kmip.server.session.00000002 - WARNING - Failure parsing request message. 2019-06-11 12:24:37,959 - kmip.server.session.00000002 - ERROR - Unrecognized attribute type: AttributeType.ALTERNATIVE_NAME Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/kmip/services/server/session.py", line 177, in _handle_message_loop request.read(request_data) File "/usr/lib/python2.7/dist-packages/kmip/core/messages/messages.py", line 354, in read batch_item.read(istream) File "/usr/lib/python2.7/dist-packages/kmip/core/messages/messages.py", line 204, in read self.request_payload.read(tstream) File "/usr/lib/python2.7/dist-packages/kmip/core/messages/payloads/create.py", line 46, in read self.template_attribute.read(tstream) File "/usr/lib/python2.7/dist-packages/kmip/core/objects.py", line 1622, in read attribute.read(tstream) File "/usr/lib/python2.7/dist-packages/kmip/core/objects.py", line 114, in read value = self.value_factory.create_attribute_value(enum_type, None) File "/usr/lib/python2.7/dist-packages/kmip/core/factories/attribute_values.py", line 108, in create_attribute_value '{0}'.format(name)) ValueError: Unrecognized attribute type: AttributeType.ALTERNATIVE_NAME 2019-06-11 12:24:37,964 - kmip.server.session.00000002 - INFO - Stopping session: 00000002

I understand this attribute is not listed in the attribute_values.py file. Is there any possibility to add support for this?

Thanks in advance

[PeterHamilton]

Hi @cocomarin, thanks for asking! This is a separate issue. You're trying to use the AlternativeName attribute, which PyKMIP doesn't support. Go ahead and file a new issue using your post text above and I'll add this to our backlog.

Adding support for the AlternativeName attribute shouldn't be too complicated (it's almost identical to the current Name attributes PyKMIP currently supports). However, given my current schedule I cannot commit to adding it in the near term. We'll use the new issue you file to track it though.

[cocomarin]

Thanks Peter, I´ll open another issue for this. Thank you very much again!

[DavitEpam]

Hi @PeterHamilton . Could you please describe briefly, in a few words, what kind of work needs to be done in PyKMIP in order to support these custom attributes? We need these custom attributes for some testing purposes. Actually we are only interested in symmetric keys. We want to add the code to support AddAttribute, ModifyAttribute and DeleteAttribute.

[PeterHamilton]

Hi @DavitEpam, sure, I can discuss that briefly. PyKMIP already supports the top-level CustomAttribute structure, but right now it's limited to only supporting string-based attributes. This was intentional when we first added CustomAttribute support long ago, since strings can generally represent all of the other primitive data types. However, complex attributes containing multiple fields require a lot more work. From a security and protocol correctness point-of-view, PyKMIP needs to know what fields to expect for each structure it decodes, so dynamically encoding/decoding a structure it doesn't recognize won't work. Furthermore, if you want to persist complex multi-field attributes in the PyKMIP server, tables and SQL updates need to be made to support that; generic string tables won't cut it. This is why in some of the prior discussion above we thought that just storing the raw encoding of the CustomAttribute would work best, since that can be stored generically. It would greatly limit the ability to filter on those attributes (since PyKMIP won't know how to parse them) but at least you could store them.

The alternative to generic storage is to actually add custom attribute implementations to PyKMIP. While possible, this isn't ideal since those attributes will only work with PyKMIP and third-party servers that come prebuilt to support those attributes. At one point I was interested in defining a mechanism to allow users and developers to define and dynamically register and add their own custom attribute implementations to the library (kind of like a PyKMIP plugin system). That way PyKMIP could support users who need custom attributes but still remain a generic solution that's compliant with the core KMIP specification. Unfortunately I've never had the chance to really dig in and explore that idea.

As for AddAttribute, ModifyAttribute, and DeleteAttribute support, hold off on doing any dev work for those. I'm currently gearing up to add those operations (or their analogs) in the next release. Those will be implemented within the next two months. I'll also be officially adding support for the ApplicationSpecificInformation attribute, which is another way to support string-based custom attributes. If you're application can get by just using string-based custom attributes (that you can arbitrarily label however you want) this may be a solution for you as well.

[DavitEpam]

Thanks for the answer @PeterHamilton . We'll wait for Add/Modify/DeleteAttribute support.

[dmrlx]

Hi, any updates here?