automatic decoding of ExtensionClasses do not support namespaces other than 0

[levkevi]

It looks like the ExtensionClasses do not support namespaces.

To add support for it, we'd need to either use a tuple as a key for ObjectIdNames and ObjectIds or add a new system for the ExtensionClasses that are not in the default namespace. What should we do? It looks like object_ids.py is autogenerated.

[zerox1212]

Did you find anything about it in the UA spec?

[bitkeeper]

ua.ObjectIdNames and ua.ObjectIds are only shorthands for ids in the default namespace 0. That is why they don't contain namespaces. The class uatypes.ExtensionObject contains a TypeId as NodeId.

So your are probably talking about something different ?

[levkevi]

Ok so I should create another data structure to support the namespaces in custom objects. I used those to register my classes but it seems that it was not the proper place to do so.

[levkevi]

Erm, I think that I should actually use a structure to return my data. I'm not exactly trying to send an object, I want to encode/decode a structure into a python object.

[zerox1212]

Are you saying you want a custom UA object to be treated as a native python object? This functionality isn't part of the server. However, if that is what you want to do I have example code you can look at which creates python objects from UA objects, then keeps them synchronized via subscription. That way in my application I only need to work with the python class.

[oroulet]

Custom structure should always be constructed as node under 0:data_types/0.Structures and to decode custom structure, we should look in the addressspace and from the node tree build automatically decode/encode. My issue is that I have not understood yet how to get the order of nodes. nodes are not ordered in a tree..... but maybe we could use the NodeId order....

so I am expected that read_attribute returned a variant containing an Extension object that we feed to a method called decode_extension object that uses the address space tree to decode and return a python object.

same for encoding

[zerox1212]

Do you know what part of the spec this might be in @oroulet ? I can take a look.

[oroulet]

No idea. It should be in mapping part but I could not find anything

[zerox1212]

Docs say it's based on the definition order. The example looks clear to me.

It's in OPC UA Spec Part6, Section 5.2.5

The way I read the spec, you have to have a custom structure defined before you add it to the tree.

[oroulet]

in the xml this is true, but in the tree, I cannot see that nodes childs have an order, we could order them by nodeid, but I am guessing this is wrong....

[oroulet]

To be able to decode unknown structures from a server, we MUST get the informations from the address space. mayeb the informations must be stored in xml in the tree, see https://github.com/FreeOpcUa/python-opcua/issues/293 which is a duplicate of this issue

[zerox1212]

From what I am reading, we have to store the definition in a DataTypeEncoding node, but I have to read some more to understand the spec is talking about...

[zerox1212]

So it appears to me that all structures will start with a TypeId. This is the node id of the DataTypeEncoding node where the definition is stored. This is how you can find out how to decode the structure. To get an actual description of this you have to jump to Part 4 of the spec and look at ReadValueId:

[oroulet]

@zerox1212 looks like you are the right path. I am afraid they want us to store the definiton as xml in the encoding node... which will be a mess to keep up to date when adding new structures. maybe we need to create a special function to create the xml or always require structures to be developed as xml and store it in encoding node after import .....

[zerox1212]

It will be tricky to keep everything up to date. The spec does mention it though:

[zerox1212]

I think we would have to make a structure_factory where you define the custom structure as lists of types. Python being based on unordered dicts will make this kind of messy. I don't think we can make an example like the C++ shown in the spec, because python objects won't serialize in the same order all the time.

[oroulet]

ou define the custom structure as lists of types.

I think a structure should be defined by creating nodes in address space. that is the entire concept of opc ua

[zerox1212]

According to the example in the spec that is not what a Custom Structure is for. It's to extend the UA (binary) protocol to communicate user data types directly. That way data can be packed/unpacked on the fly, similar to how GUID is working. That is how I understand it.

[oroulet]

Did I write something else?

On Tue, Nov 22, 2016, 20:05 Andrew notifications@github.com wrote:

According to the example in the spec that is not what a Custom Structure is for. It's to extend the UA (binary) protocol to communicate user data types directly. That way data can be packed/unpacked on the fly, similar to how GUID is working.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/FreeOpcUa/python-opcua/issues/373#issuecomment-262334078, or mute the thread https://github.com/notifications/unsubscribe-auth/ACcfzqDoiRaYdo70aGea4jWuxAuHr5qqks5rAz0GgaJpZM4K4qD7 .

[levkevi]

So, if I get this right:

• When calling a method, the client must send a Variant of ExtensionObjects to the server
• The server also return a Variant of ExtensionObjects

In the ExtensionObject, the body is encoded using a ByteString or a XmlElement, depending on the Encoding used (Mapping (part 6), 5.2.2.15)

• The first element of the ExtensionObject body is the Type Id (I suppose it should be a NodeId), which refer to the Structured DataType of the data it contains. (Mapping (part 6), 5.2.5)

[oroulet]

Yes. One issue is that this is usually not a variant of ExtensionObjects but a variant of a python object, which is converted to ExtensionObjects in the function ExtensionObjects _to_binary. This function uses a hard coded dict of TypeId:pythonobjects. We could register our objects in there, but for a client, we cannot register, we need to get that information from the address space.....

On Tue, Nov 22, 2016, 20:20 levkevi notifications@github.com wrote:

So, if I get this right:

• When calling a method, the client must send a Variant of ExtensionObjects to the server
• The server also return a Variant of ExtensionObjects

In the ExtensionObject, the body is encoded using a ByteString or a XmlElement, depending on the Encoding used (Mapping (part 6), 5.2.2.15)

• The first element of the ExtensionObject body is the Type Id (I suppose it should be a NodeId), which refer to the Structured DataType of the data it contains. (Mapping (part 6), 5.2.5)

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/FreeOpcUa/python-opcua/issues/373#issuecomment-262338029, or mute the thread https://github.com/notifications/unsubscribe-auth/ACcfzofWg6Vmu65sH8nR2WzX8UimDLltks5rA0BigaJpZM4K4qD7 .

[zerox1212]

Did I write something else?

My point was that I'm not reading anything that this structure actually needs to be defined with nodes. Maybe I don't understand the spec, but it looks like you can have a custom structure that is only defined by the data in a DataTypeEncoding node (which would come from a custom python object for example). That structure can be found by Has Encoding reference.

However I can be wrong here, because I'm still trying to understand all the ExtensionObject stuff.

[zerox1212]

After thinking about this... does the spec intend that every extension object have a DataTypeEncoding node? And then if a client that doesn't know about the extension object attempt to read one that it has to add the ExtensionObject to it's address space based on the data in DataTypeEncoding?

[oroulet]

I think we need both the data structure under data types/ structure and the xml in the encoding node( there is only one as I understood)

On Tue, Nov 22, 2016, 21:17 Andrew notifications@github.com wrote:

After thinking about this... does the spec intend that every extension object have a DataTypeEncoding node? And then if a client that doesn't know now about the extension object attempt to read one that it has to add the ExtensionObject to it's address space based on the data in DataTypeEncoding?

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/FreeOpcUa/python-opcua/issues/373#issuecomment-262353586, or mute the thread https://github.com/notifications/unsubscribe-auth/ACcfzunktf0IjtqYxQ04tm3cxL-htXdtks5rA03XgaJpZM4K4qD7 .

[zerox1212]

Then python opcua doesn't fully support extension objects at the moment? The only way the current python client could read/write an extension object on python opcua server is if the extension object was already defined manually in the client. Is that correct?

[bitkeeper]

If you create a xml UANodeSet schemawith UaModeler it give some clues how the struct description can be stored in regular nodes. Also an xsd an bsd file are created.

schema.zip

[oroulet]

Currently you need to do manual encoding/decoding

On Tue, Nov 22, 2016, 21:29 Andrew notifications@github.com wrote:

Then python opcua doesn't fully support extension objects at the moment? The only way the current python client could read/write an extension object on python opcua server is if the extension object was already defined manually the client. Is that correct?

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/FreeOpcUa/python-opcua/issues/373#issuecomment-262356543, or mute the thread https://github.com/notifications/unsubscribe-auth/ACcfzs4F8VQ_3sGwTO_-2MFQz-53SNvsks5rA1CfgaJpZM4K4qD7 .

[levkevi]

It looks like the DataTypeEncoding is defined in Address SPace Model (part 3) 5.8.4

[levkevi]

Btw guys, Python has an OrderedDict class in the standard library:

https://docs.python.org/2/library/collections.html#collections.OrderedDict

[oroulet]

As also mentionned in https://github.com/FreeOpcUa/python-opcua/issues/293 It is now clear that structures are not defines as a set of node (as they should in my opinion) but as xml under the Types/ Data Types/OPC Binary node.

I have started something there: https://github.com/FreeOpcUa/python-opcua/tree/extobj but when I think about it, it might be the wrong approach. generating classes on the fly, is very flexible, maybe sometime necesary but makes debugging very hard.

So on client side I am thinking about adding a method to make it easy to download xml structtures specifications from server. then make a script to generate a .py file containing the structures (most code already exist to generate protocol code). As @levkevi proposed we can then add a method to register these classes for automatic decoding from binary

On server side we need to define these structures. We can add a simple function that takes a list of variables and uatypes and generate an xml file which can then be used, as in client, to generate a .py file. We then also need a function to register these classes/structures on server side (create nodes, add xml under ua_binary node, and enable automatic decoding)

crazy complicated but that should work...

[zerox1212]

I think that if you do it that way it will be difficult to implement StructureChanged feature. Would you regenerate the entire encoding py file if the server changed the data type?

Seems like it would be better to do it on the fly, if the client runs into an extension object, it checks a cache for the encoding node before reading, if the encoding isn't found it gets the encoding from the server, caches the encoding somewhere, then continues to decode the extension object. This allows us to also check the server for StructureChanged and update our cached encoding.

I think this is what OPC spec intends. True it might be difficult to debug, but I think it will not be that bad because user will simply look at the encoding node it gets from the server to see what his python object will be.

[bitkeeper]

I also supports the idea of on the fly without a generator like @zerox1212 suggests.

@oroulet is it really necessary to have a API at server side to describe an extobj? The spec already provide something for that; a XML XSD. For example with UaModeller you can already play arround with it, and just load the nodes.

btw xml handling is a lotter easier with a decent library like lxml instead of the default etree.

[zerox1212]

We used etree to limit external dependencies. It's @oroulet call if you want to use a different XML library. Someone would also need to refactor it which might be a major undertaking.

[oroulet]

on-the-fly class generation is what I implemented in #293 but really it makes debugging very hard and break code compleation, code checkers, etc.... Another possibility is to to generate a file on the fly and import it. It sounds very hacky but at least we can keep the generated py file for code completion, debugging and code checker.

@bitkeeper if you can show that lxml simplifies a lot the import code, then I am fine with adding a dependency. Code export is already quite clean, so I am not sure it is possible to improve it a lot...

@bitkeeper I am not thinking of adding any special API, we will use the xml stuff, but maybe a function to generate thal xml from code. We will still support importing xml file anyway

[zerox1212]

Can we make the classes on the fly by default, but have an option to write the classes (encodings) to a file for purpose of debugging or code completion?

In my example you would just add a step: when client gets an extension object, it first checks if it's in the py file that was imported, if the class isn't there we make the class on the fly and cache the encoding. That way it will always work with any OPC UA server with out special steps, but we also give the developer flexibility to predefine the objects for code completion/debugging.

Anyways, wouldn't it be complicated for the client to handle a extension object that the server has changed? You would have to remove the import, then import it again.. or modify the imported class in place. This might be worse than the debugging issues from on the fly creation.

I likely won't be working on this, so these are just ideas. Whoever has time to implement this will get to decide. :)