search

FreeOpcUa / python-opcua

LGPL Pure Python OPC-UA Client and Server
http://freeopcua.github.io/
GNU Lesser General Public License v3.0
1.37k stars 660 forks source link

Import scheme of custom datatypes #221

Closed bitkeeper closed 8 years ago

bitkeeper commented 8 years ago

I trying to import fragment below with XmlImporter.import_xml. It contains a custom datatype.

  <UADataType NodeId="ns=1;i=3008" BrowseName="1:MyCustomString">
    <DisplayName>MyCustomString</DisplayName>
      <References>
          <Reference ReferenceType="HasSubtype" IsForward="false">i=12</Reference>
    </References>
  </UADataType>

 <UAVariable NodeId="i=30007" BrowseName="MyCustomTypeVar" DataType="MyCustomString">
    <References>
      <Reference ReferenceType="HasTypeDefinition">i=69</Reference>
      <Reference ReferenceType="Organizes" IsForward="false">i=30002</Reference>
    </References>
  </UAVariable>

It fails with:

  File "D:\work\python-opcua\tests\tests_server.py", line 132, in test_xml_import
    self.srv.import_xml("tests/custom_nodes.xml")
  File ".\opcua\server\server.py", line 370, in import_xml
    importer.import_xml(path)
  File ".\opcua\common\xmlimporter.py", line 28, in import_xml
    self.add_variable(node)
  File ".\opcua\common\xmlimporter.py", line 98, in add_variable
    attrs.DataType = self.to_data_type(obj.datatype)
  File ".\opcua\common\xmlimporter.py", line 59, in to_data_type
    return ua.NodeId(getattr(ua.ObjectIds, nodeid))
AttributeError: type object 'ObjectIds' has no attribute 'MyCustomString'

It looks the to_data_type is limited to using types of the standard address space. Ok no problem. Now I want add support for custom types, could you suggest the preferred way to get the nodeid:

zerox1212 commented 8 years ago

FYI the XML importer only offers basic functionality. Please feel free to work on it. :)

I don't know if making a custom datatype is currently supported in the code. Perhaps you could look at how custom events nodes are created and follow a similar model? Custom events are built from the BaseEventType, however they don't add anything to ObjectIds.

My guess is that custom ObjectIds would be the responsibility of the user to implement when customizing thier OPC UA server.

Can I ask why you need a custom datatype? I never really understood the use case for this feature.

oroulet commented 8 years ago

I implemented the parser and I confirm that my goal was to import the standard address space so it might be missing features and the code is quite ugly... I haven't thought much about it but I am against extending the objectids class. This is already a performance bottleneck.... Maybe traversing the node structure is ok. This is a one time operation anyway. , try and let us know

On Tue, Jun 21, 2016, 20:33 Andrew notifications@github.com wrote:

FYI the XML importer only offers basic functionality.

I don't know if making a custom datatype is currently supported in the code. Perhaps you could look at how custom events nodes are created and follow a similar model? Custom events are built from the BaseEventType, however they don't add anything to ObjectIds.

My guess is that custom ObjectIds would be the responsibility of the user to implement when customizing thier OPC UA server.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/FreeOpcUa/python-opcua/issues/221#issuecomment-227530590, or mute the thread https://github.com/notifications/unsubscribe/ACcfztAqz3trdeZv3TTclfd1RgYz747qks5qOC5rgaJpZM4I64z0 .

bitkeeper commented 8 years ago

@zerox1212 : Defining the and importing custom datatype isn't a problem at all. The problem is using it because the only a lookup (from datatype name to nodeid ) in standard namespace is supported.

Typical use cases are own enumerations and structs. I would like to use a tool like UXModeler to design an address space, including custom object, data, list and event types (have now a working handcrafted alarm proto) .

@oroulet : Ok I will give the node traverse a try.

zerox1212 commented 8 years ago

@bitkeeper It would be great if you could share your tool. I was thinking of the same thing and I started implementing it. However I didn't have time so I just have a python script that builds most of my address space in XML, then I have to edit a few things by hand (which sucks).

The next step I was going to tackle was to export the address space to XML so you could have a nice file based backup/restore option. Instead I just configure my address space once then dump it to a binary. This works but can't be edited.

bitkeeper commented 8 years ago

@zerox1212 The modeling tool tool UaModeler just comes from Unfied Automation.

oroulet commented 8 years ago

@bitkeeper does uamodeller exports the same XML format as the spec uses?

bitkeeper commented 8 years ago

@oroulet yes no problem

bitkeeper commented 8 years ago

Guys you review the following:

A nice place for fixing support for custom datatypes seams the function XmlImporter.to_data_type:

   def to_data_type(self, nodeid):
        if not nodeid:
            return ua.NodeId(ua.ObjectIds.String)
        if "=" in nodeid:
            return ua.NodeId.from_string(nodeid)
        elif hasattr(ua.ObjectIds, nodeid):     # changed to check if nodeid is present
            return ua.NodeId(getattr(ua.ObjectIds, nodeid))
        else:                                     
            return self.find_type(nodeid)        # else find it by browse the nodes

First I tried to implement find_type (as standalone function ) like:

    def find_type(node, type_to_find, nodeclassmask = ua.ObjectIds.Organizes):
        match = None
        for child in node.get_children():
            if child.get_browse_name().Name == type_to_find:
                match = child
                break
            else:
                match = find_type(child, type_to_find, nodeclassmask = ua.ObjectIds.HasSubtype)
                if match:
                    break
        return match

    nodeid = find_type(server.get_root_node().get_child(["0:Types"]), "MyObjectType")

That works fine if I use it outside the XmlImporter, but it seems that the nodes inside XmlImporter contains a different structure and have a different type of object for the attribute server.

Which resulted in the following member function of XmlImporter:

    def find_type(self, type_to_find):
        """
        Finds NodeIs of Type based on the BrowseName.name.

        :param type_to_find: string of the type to find.
        :returns NodeId of found type_of_find
        :raises AttributeError: In case type is not found
        """        
        match = None

        for childid in self.server._aspace.keys():
            n = self.server._aspace[childid]
            for ref in n.references:
                if ref.NodeClass & (NodeClass.ObjectType + NodeClass.VariableType +  NodeClass.DataType):                    
                    if ref.BrowseName.Name == type_to_find:
                        match = childid
                        break
            else:
                continue
            break

        return ua.NodeId(match.Identifier, match.NamespaceIndex)

Is this the way to go ?

oroulet commented 8 years ago

First a question: Is is correct that a datatype can be specifiied by a truncated qualified name without namespace? that looks like a really broken thing... yes to_data_type looks like the good place :-)

I see that we only passe the node_mgt service to xmlimporter, you should change this to pass the iserver object and update code accordingly. then you will have access to all methods. and can create a Node object Node(self.iserver, nodeid) then use method1 starting from DataType node (not Root node) to browse for all exixting type. method2 is broken since it addresse internal data without locking...

bitkeeper commented 8 years ago

About the using the qname without namespace:

I can only find examples of xml address space where the value of datatype is without namespace:

 <UAVariable NodeId="i=30007" BrowseName="MyCustomTypeVar" DataType="MyCustomString">

Also UAModeler generates it without the namespace. The XSD of the address space format can not clarify this because it can only indicate the type of value, not the content of the string.

Maybe the datatypes should be first resolved in the alias section of xml, the aliases does contain the namespaces:

    <Aliases>
        ....
        <Alias Alias="MyType1">ns=1;i=3002</Alias>
        <Alias Alias="MyType2">ns=1;i=3003</Alias>
        <Alias Alias="MyType3">ns=1;i=3004</Alias>
    </Aliases>

Any experience with this ?

oroulet commented 8 years ago

Using alias make sense!, much more than qname without index. How do you generate XML files? Do they come without alias?

On Tue, Jun 28, 2016, 22:50 Marcel notifications@github.com wrote:

About the using the qname without namespace:

I can only find examples of xml address space where the value of datatype is without namespace:

Also UAModeler generates it without the namespace. The XSD of the address space format can not clarify this because it can only indicate the type of value, not the content of the string. Maybe the datatypes should be first resolved in the alias section of xml, the aliases does contain the namespaces: ``` .... ns=1;i=3002 ns=1;i=3003 ns=1;i=3004 ``` Any experience with this ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/FreeOpcUa/python-opcua/issues/221#issuecomment-229179050, or mute the thread https://github.com/notifications/unsubscribe/ACcfziildjl916XsCgmBpx3nt9Er_dmpks5qQYkYgaJpZM4I64z0 .
bitkeeper commented 8 years ago

I use UaModeler from Unified Automation and it generates also the aliases.

Ok so lets support the aliases, I think also that this is the intention. The XmlImporter already contains a function to_ref_type to lookup the nodeid of an alias. This makes it easy to integrate the alias support into XmlImporter

    def to_data_type(self, nodeid):
        if not nodeid:
            return ua.NodeId(ua.ObjectIds.String)
        if "=" in nodeid:
            return ua.NodeId.from_string(nodeid)
        elif hasattr(ua.ObjectIds, nodeid):
            return ua.NodeId(getattr(ua.ObjectIds, nodeid))
        elif ":" in nodeid:  # add support for full qnames 
            return self.find_qname(nodeid) # new function
        else: # added support for aliases
            return self.to_ref_type(nodeid) # function already present

Which lead to support of:

Good enough for me.

Example implementation of XmlImporter.find_qname, still based on self.server is node_mgt service. Rewriting the XmlImporter is a different issue, maybe performed by someone with a little more knowledge about the internals then me ..

    def find_qname(self, qname_to_find):
        """Finds NodeIs of Type based on the QName.
        :param type_to_find: string of the qname to find. Like '1:MyType'
        :returns NodeId of found qname_to_find
        :raises AttributeError: In case type is not found
        """
        match = None
        ns, name = qname_to_find.split(':')

        for childid in self.server._aspace.keys():
            n = self.server._aspace[childid]
            for ref in n.references:
                if ref.NodeClass & (NodeClass.ObjectType + NodeClass.VariableType + NodeClass.DataType):
                    if ref.BrowseName.to_string() == qname_to_find:  # and ref.BrowseName.NamespaceIndex':
                        match = childid
                        break
            else:
                continue
            break

        return ua.NodeId(match.Identifier, match.NamespaceIndex)
oroulet commented 8 years ago

Do you really need the qname stuff? it looks very strange, but it would be great if you could send a pull request for the alias stuff

zerox1212 commented 8 years ago

UA Modeler looks nice. I'm testing it now.

The only thing I don't like is that String NodeIds will be pain to use because it won't create them automatically. It only auto generates integer NodeIds. I prefer strings because they are way more readable.

One thing I did notice. If you make a variable in UA Modeler and give it a value the XML importer might have problems with the "Value" format.

<Value>

<uax:Float>100</uax:Float>

</Value>

The python example XML does not have "uax:".

I'm looking forward to a PR.

bitkeeper commented 8 years ago

@oroulet : I don't think it is needed:

The xsd (https://opcfoundation.org/UA/2011/03/UANodeSet.xsd) indicates that datatype is a nodeid.

<xs:complexType name="UAVariable">
    <xs:complexContent>
      <xs:extension base="UAInstance">
        <xs:sequence>
          <xs:element name="Value" minOccurs="0">
            <xs:complexType>
              <xs:sequence>
                <xs:any minOccurs="0" processContents="lax" />
              </xs:sequence>
            </xs:complexType>
          </xs:element>
          <xs:element name="Translation" type="TranslationType" minOccurs="0" maxOccurs="unbounded"></xs:element>
        </xs:sequence>
        <xs:attribute name="DataType" type="NodeId" default="i=24"></xs:attribute>
        <xs:attribute name="ValueRank" type="ValueRank" default="-1"></xs:attribute>
        <xs:attribute name="ArrayDimensions" type="ArrayDimensions" default=""></xs:attribute>
        <xs:attribute name="AccessLevel" type="AccessLevel" default="1"></xs:attribute>
        <xs:attribute name="UserAccessLevel" type="AccessLevel" default="1"></xs:attribute>
        <xs:attribute name="MinimumSamplingInterval" type="Duration" default="0"></xs:attribute>
        <xs:attribute name="Historizing" type="xs:boolean" default="false"></xs:attribute>
      </xs:extension>
    </xs:complexContent>
  </xs:complexType>

Which I think indicates that if name is used it is always an alias. Even the used basic types like string are mentioned in the alias list.

So lets get rid of the qname function.