get_child - BadNoMatch issue accessing an array in a Siemens OPC UA Server

[itrukw]

Hello, We are using the python-opcua library (0.90.3) to access a Siemens OPC UA server (15xx PLC / TIA). Everything is working fine so far, but if we like to access a node which is an array in the Siemens DB we do get a BadNoMatch error. You can see here the structure within the UaExpert: The BrowseName of the first array element is shown as 3,"0". We try to access e.g. the alarm_id of OP 0 and we tried to address it in the following way:

root = client.get_root_node()
alarm_ID = root.get_child( ["0:Objects", "3:FGDKD", "3:DataBlocksGlobal", "3:Comm_PLC_Nemetris_JobView", "3:OP", "3:0", "3:alarm_id"] )

The requested operation has no match to return.(BadNoMatch)`

Reading until OP works fine (["0:Objects", "3:FGDKD", "3:DataBlocksGlobal", "3:Comm_PLC_Nemetris_JobView", "3:OP"]), but as soon as we like to access one of the array elements it fails. Do we do something wrong building the browse path, or is there an issue with accessing this special structure?

Thx

[oroulet]

As far as I can see, OP is a variable node. The strange thing is that they placed variables under variables. This is not usual but possible. But they may have used a strange link between these variables. What kind of links do you have from OP to its children? (You can see it somewhere in uaexpert)

[itrukw]

Hi, Thanks for the response.

At first I show you the the Siemens definition:

The Siemens OPC UA server shows the following structure for the OP:

OP is somehow an array of Nem.StationInfo

In the DataTypes section of the OPC UA Server there is a node SimanticStructures and there you just can see Nem.StationInfo and all of its childs (Nem.StationInfo.alarm_id, etc.). OP itself is not listed because it is an array of Nem.StationInfo?!

Do you have any idea what the issue is? Do you need more information?

By the way .... Siemens uses for any struct or array a variable node, so you have a lot of nested variable nodes in the OPC UA server because you often use nested structs while dealing with a Siemens PLC programm. Structs are not a problem with your Client implementation but somehow this array thing drives me nuts ... Siemens is pushing the OPC UA server on their CPU but their complex data types are "complex". Writing on such a complex data type will be the next challenge which we need to do. We have faced so far a lot of issues with the Siemens OPC UA server. If you know people who are also dealing with the Siemens world, we can share our experiance.

[oroulet]

Ok then maybe this is an array. Call get_value() on OP node. Do you get a python list? Of what? Try to call a method on client object called something like 'import and registrer' before

[itrukw]

Hi, Yes get_value() of the OP node returns a list of ExtensionObjects:

get_browse_name() and get_value() for the OP node:

BrowseName: QualifiedName(3:OP)
Value:
[
  ExtensionObject(TypeId:StringNodeId(ns=3;s=TE_"Nem.StationInfo"), Encoding:1, 44 bytes),
  ExtensionObject(TypeId:StringNodeId(ns=3;s=TE_"Nem.StationInfo"), Encoding:1, 44 bytes),
  ExtensionObject(TypeId:StringNodeId(ns=3;s=TE_"Nem.StationInfo"), Encoding:1, 44 bytes),
  ExtensionObject(TypeId:StringNodeId(ns=3;s=TE_"Nem.StationInfo"), Encoding:1, 44 bytes),
  ExtensionObject(TypeId:StringNodeId(ns=3;s=TE_"Nem.StationInfo"), Encoding:1, 44 bytes),
  ExtensionObject(TypeId:StringNodeId(ns=3;s=TE_"Nem.StationInfo"), Encoding:1, 44 bytes),
  ExtensionObject(TypeId:StringNodeId(ns=3;s=TE_"Nem.StationInfo"), Encoding:1, 44 bytes),
  ExtensionObject(TypeId:StringNodeId(ns=3;s=TE_"Nem.StationInfo"), Encoding:1, 44 bytes),
  ExtensionObject(TypeId:StringNodeId(ns=3;s=TE_"Nem.StationInfo"), Encoding:1, 44 bytes),
  ExtensionObject(TypeId:StringNodeId(ns=3;s=TE_"Nem.StationInfo"), Encoding:1, 44 bytes),
  ExtensionObject(TypeId:StringNodeId(ns=3;s=TE_"Nem.StationInfo"), Encoding:1, 44 bytes),
  ExtensionObject(TypeId:StringNodeId(ns=3;s=TE_"Nem.StationInfo"), ...
  ExtensionObject(TypeId:StringNodeId(ns=3;s=TE_"Nem.StationInfo"), Encoding:1, 44 bytes),
  ExtensionObject(TypeId:StringNodeId(ns=3;s=TE_"Nem.StationInfo"), Encoding:1, 44 bytes),
  ExtensionObject(TypeId:StringNodeId(ns=3;s=TE_"Nem.StationInfo"), Encoding:1, 44 bytes
]

-----------------------------------------------------------------------------------------------
Walking through the child nodes with get_browse_name() and get_value()

    QualifiedName(3:0) - Value: ExtensionObject(TypeId:StringNodeId(ns=3;s=TE_"Nem.StationInfo"), Encoding:1, 44 bytes)
      QualifiedName(3:op_status) - Value: ExtensionObject(TypeId:StringNodeId(ns=3;s=TE_"Nem.StationInfo"."op_status"), Encoding:1, 16 bytes)
        QualifiedName(3:bit1) - Value: False
        QualifiedName(3:bit2) - Value: False
        QualifiedName(3:bit3) - Value: False
        QualifiedName(3:bit4) - Value: False
        QualifiedName(3:bit5) - Value: False
        QualifiedName(3:bit6) - Value: False
        QualifiedName(3:bit7) - Value: False
        QualifiedName(3:bit8) - Value: False
        QualifiedName(3:bit9) - Value: False
        QualifiedName(3:bit10) - Value: False
        QualifiedName(3:bit11) - Value: False
        QualifiedName(3:bit12) - Value: False
        QualifiedName(3:bit13) - Value: False
        QualifiedName(3:bit14) - Value: False
        QualifiedName(3:bit15) - Value: False
        QualifiedName(3:bit16) - Value: False
      QualifiedName(3:alarm_id) - Value: 0
      QualifiedName(3:control_bits) - Value: ExtensionObject(TypeId:StringNodeId(ns=3;s=TE_"Nem.StationInfo"."control_bits"), Encoding:1, 16 bytes)
        QualifiedName(3:bit1) - Value: False
        QualifiedName(3:bit2) - Value: False
        QualifiedName(3:bit3) - Value: False
        QualifiedName(3:bit4) - Value: False
        QualifiedName(3:bit5) - Value: False
        QualifiedName(3:bit6) - Value: False
        QualifiedName(3:bit7) - Value: False
        QualifiedName(3:bit8) - Value: False
        QualifiedName(3:bit9) - Value: False
        QualifiedName(3:bit10) - Value: False
        QualifiedName(3:bit11) - Value: False
        QualifiedName(3:bit12) - Value: False
        QualifiedName(3:bit13) - Value: False
        QualifiedName(3:bit14) - Value: False
        QualifiedName(3:bit15) - Value: False
        QualifiedName(3:bit16) - Value: False
      QualifiedName(3:work_instruction) - Value: ExtensionObject(TypeId:StringNodeId(ns=3;s=TE_"Nem.StationInfo"."work_instruction"), Encoding:1, 6 bytes)
        QualifiedName(3:job_sequence) - Value: 0
        QualifiedName(3:module_id) - Value:
      QualifiedName(3:plc_live_text) - Value:
    QualifiedName(3:1) - Value: ExtensionObject(TypeId:StringNodeId(ns=3;s=TE_"Nem.StationInfo"), Encoding:1, 44 bytes)
      QualifiedName(3:op_status) - Value: ExtensionObject(TypeId:StringNodeId(ns=3;s=TE_"Nem.StationInfo"."op_status"), Encoding:1, 16 bytes)
        QualifiedName(3:bit1) - Value: False
        QualifiedName(3:bit2) - Value: False
        QualifiedName(3:bit3) - Value: False
        QualifiedName(3:bit4) - Value: False
        QualifiedName(3:bit5) - Value: False
        QualifiedName(3:bit6) - Value: False
        QualifiedName(3:bit7) - Value: False
        QualifiedName(3:bit8) - Value: False
        QualifiedName(3:bit9) - Value: False
        QualifiedName(3:bit10) - Value: False
        QualifiedName(3:bit11) - Value: False
        QualifiedName(3:bit12) - Value: False
        QualifiedName(3:bit13) - Value: False
        QualifiedName(3:bit14) - Value: False
        QualifiedName(3:bit15) - Value: False
        QualifiedName(3:bit16) - Value: False
      QualifiedName(3:alarm_id) - Value: 0
      QualifiedName(3:control_bits) - Value: ExtensionObject(TypeId:StringNodeId(ns=3;s=TE_"Nem.StationInfo"."control_bits"), Encoding:1, 16 bytes)
        QualifiedName(3:bit1) - Value: False
        QualifiedName(3:bit2) - Value: False
        QualifiedName(3:bit3) - Value: False
        QualifiedName(3:bit4) - Value: False
      [...]

[oroulet]

OK. then you have an array. We need to make client understand that custom array format. What happens if you call import_and_register_structures() before get_value()? This method is only available in master version of python-opcua in github I think.

[itrukw]

OK, I have tried it with the current master, but I do get an exception:

root = client.get_root_node()
client.import_and_register_structures() # scan server for custom structures and import them

No module named 'structures_Opc'
Traceback (most recent call last):
  File "opc_ua_test.py", line 61, in <module>
    client.import_and_register_structures()  # scan server for custom structures and import them
  File "/opt/.../lib/python3.4/site-packages/opcua/client/client.py", line 579, in import_and_register_structures
    gen.save_and_import(name + ".py", append_to=structs_dict)
  File "/opt/.../lib/python3.4/site-packages/opcua/common/structures_generator.py", line 178, in save_and_import
    mymodule = importlib.import_module(name)
  File "/opt/python34/lib/python3.4/importlib/__init__.py", line 109, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 2254, in _gcd_import
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2212, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 2254, in _gcd_import
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2212, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 2254, in _gcd_import
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2224, in _find_and_load_unlocked
ImportError: No module named 'structures_Opc'

[oroulet]

was .py file created in you current directory? should be one called structures_Opc.py

[itrukw]

There was just one file generated: structures_Opc.Ua.Di.py

I guess you try to read the DataTypes structure. With uaexpert I see the following types. The SimaticStructures are the Siemens data types.

client.nodes.opc_binary.get_children_descriptions() shows also this result:

[
ReferenceDescription(ReferenceTypeId:TwoByteNodeId(i=47), IsForward:True, NodeId:FourByteNodeId(i=7617),      BrowseName:QualifiedName(0:Opc.Ua), DisplayName:LocalizedText(Encoding:2, Locale:None, Text:b'Opc.Ua'), NodeClass:NodeClass.Variable, TypeDefinition:TwoByteNodeId(i=72)),
ReferenceDescription(ReferenceTypeId:TwoByteNodeId(i=47), IsForward:True, NodeId:FourByteNodeId(ns=2;i=6435), BrowseName:QualifiedName(2:Opc.Ua.Di), DisplayName:LocalizedText(Encoding:2, Locale:None, Text:b'Opc.Ua.Di'), NodeClass:NodeClass.Variable, TypeDefinition:TwoByteNodeId(i=72)),
ReferenceDescription(ReferenceTypeId:TwoByteNodeId(i=47), IsForward:True, NodeId:FourByteNodeId(ns=3;i=6001), BrowseName:QualifiedName(3:SimaticStructures), DisplayName:LocalizedText(Encoding:3, Locale:b'en-US', Text:b'SimaticStructures'), NodeClass:NodeClass.Variable, TypeDefinition:TwoByteNodeId(i=72))
]

Opc.Ua is excluded, so there should be 2 files!?

I did a little test and debugging and at least to this point it looks ok:

[Node(FourByteNodeId(ns=2;i=6435)), Node(FourByteNodeId(ns=3;i=6001))]
structures_Opc.Ua.Di
structures_SimaticStructures

[oroulet]

yes the client should read the value of this nodes, parse it, create two .py files then import them. Can you try to look at what happens? add some prints statements

[oroulet]

btw what is the value of these nodes? text or binary?

[itrukw]

Hi, I have found the first issue. The problem is that the name of the node has dots in: This leads to a file name structures_Opc.Ua.Di.py mymodule = importlib.import_module(name) ends up to just import structures_Opc and this is not existing.

I did a workaround by replacing dots with underline to get a valid name

name = name.replace(".", "_")
gen.save_and_import(name + ".py", append_to=structs_dict)

With this fix I have reached the Siemens custom objects, but there I have faced the next issue ...

Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.

The generated XML looks like this and I think this is what lxml does not like:

<?xml version="1.0" encoding="utf-8"?><opc:TypeDictionary
DefaultByteOrder="LittleEndian"
TargetNamespace="http://www.siemens.com/simatic-s7-opcua"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:opc="http://opcfoundation.org/BinarySchema/"
xmlns:ua="http://opcfoundation.org/UA/"
xmlns:tns="http://www.siemens.com/simatic-s7-opcua"
><opc:Import Namespace="http://opcfoundation.org/UA/"/>
<opc:StructuredType BaseType="ua:ExtensionObject" Name="&quot;Bockdata_1&quot;.&quot;Bockdaten&quot;">
<opc:Field TypeName="opc:Int32" Name="JobData_Size"/>
<opc:Field TypeName="tns:JobData_ATM_Test" Name="JobData" LengthField="JobData_Size"/>
<opc:Field TypeName="opc:Int32" Name="ResultData_Size"/>
<opc:Field TypeName="tns:Data_backflash_ATM_1" Name="ResultData" LengthField="ResultData_Size"/>
</opc:StructuredType>
<opc:StructuredType BaseType="ua:ExtensionObject" Name="Bockdata_1">
<opc:Field TypeName="tns:&quot;Bockdata_1&quot;.&quot;Bockdaten&quot;" Name="Bockdaten"/>
</opc:StructuredType>
<opc:StructuredType BaseType="ua:ExtensionObject" Name="Data_backflash_ATM_1">
<opc:Field TypeName="opc:Float" Name="value_A_01"/>
...

I have removed with another fast hack the encoding information: xml = xml.replace('<?xml version="1.0" encoding="utf-8"?>', "") This worked for the lxml parser.

This brought me again to the next issue. Siemens is using dots in their variable names which leads to an invalid genereated source code:

I guess this will screw up large parts of your ideas with import_and_register_structures()?!

[itrukw]

I forgot the node type:

[zerox1212]

The ability to auto generate extension objects as python object is very new so bug fixes are welcome.

There was much discussion about how to handle extension objects in Python OPC UA and creating modules to make the definitions was the direction it went. It is not all that easy to implement extension objects. In your case it might be better to pre-define the extension objects your client will use by hand.

[itrukw]

I totally agree and Siemens looks here very special. We already did some pre-defining to get complex data out of the Siemens OPC UA server. Nevertheless the journey with import_and_register_structures() was just a way to check if the initial problem with accessing the child of the strange array could be solved. It does not, so I need to check again way get_child() has an issue addressing such a Siemens array.

[oroulet]

@itrukw thanks for your efforts debuging this, this really helps make the stack reliable. Should not be very hard to fix. If everything goes right I should get a small project next week to work exactly on that kind of things. Great if you can help testing at that point.

[zerox1212]

Leave it to Siemens to create such a stupid naming convention.

[itrukw]

So we have figured out more details:

alarm_ID = root.get_child( ["0:Objects", "3:DATA_PLC", "3:DataBlocksGlobal", "3:Comm_PLC_Nemetris_JobView", "3:OP", "3:0", "3:alarm_id"] )
==> this does not work and I think there is nothing to do to make this working

alarm_ID = client.get_node('ns=3;s="Comm_PLC_Nemetris_JobView"."OP"[0]."alarm_id"')
==> this works

With this knowledge I can continue to work. btw we also found that other OPC UA implentations has issues with arrays in ExtensionObjects. This seams not to be easy stuff. @oroulet We will also plan working on an automatic generated solution for Siemens ExtensionObjects. We have checked your import_and_register_structures way and we try to have it working for Siemens, but this needs to wait a little. We need to finish other projects first but we will support you to make the library usable for the Siemens world. If you like to have something tested with an OPC UA server on a Siemens PLC, just let me know.

[oroulet]

Hi, Do you still have that server available? Can you test with https://github.com/FreeOpcUa/python-opcua/pull/494 That should work, and if it does not I would really like to know

[itrukw]

Hi, yes the server is available and I will tell one of my team members to do the testing.

[mpf82]

Hello,

I could not get the latest version to work out-of-the-box, because I could not get lxml to install/run in our Windows environment.

Instead, I decided to replace the lxml functionality with xmltodict in structures_generator.py:

#from lxml import objectify
import xmltodict

def make_model_from_string(self, xml):
    #obj = objectify.fromstring(xml)
    obj = xmltodict.parse(xml, dict_constructor=dict, attr_prefix='', force_list=['opc:Field'])
    self._make_model(obj)

def _make_model(self, root):
    root_st = root.get("opc:TypeDictionary", {}).get("opc:StructuredType", [])
    for child in root_st:
        struct = Struct(child.get("Name"))
        array = False
        for xmlfield in child.get("opc:Field", []):
            name = xmlfield.get("Name")
            if name.startswith("NoOf"):
                array = True
                continue
            field = Field(name)
            field.uatype = xmlfield.get("TypeName")
            if ":" in field.uatype:
                field.uatype = field.uatype.split(":")[1]
            field.value = get_default_value(field.uatype)
            if array:
                field.array = True
                field.value = []
                array = False
            struct.fields.append(field)
        self.model.append(struct)

I also had to change import_and_register_structures() in client.py to make sure no . are in the filename, otherwise importing the generated module fails:

        name = "structures_" + node.get_browse_name().Name
        name = name.replace(".", "_")

However, the code still generates invalid classes:

class "Bockdata_1"."Bockdaten"(object):

Probably because the child node of StructuredType already looks like this:

{
    'BaseType': 'ua:ExtensionObject',
    'Name': '"Bockdata_1"."Bockdaten"',
    'opc:Field': [{
        'TypeName': 'opc:Int32',
        'Name': 'JobData_Size'
    }, {
        'TypeName': 'tns:JobData_ATM_Test',
        'Name': 'JobData',
        'LengthField': 'JobData_Size'
    }, {
        'TypeName': 'opc:Int32',
        'Name': 'ResultData_Size'
    }, {
        'TypeName': 'tns:Data_backflash_ATM_1',
        'Name': 'ResultData',
        'LengthField': 'ResultData_Size'
    }]
}

Please let me know if you need further information. Maybe you could also consider to get rid of the lxml requirement?

Cheers, Mike

[oroulet]

You need to test the linked PR, not master as the code is not merged yet. The method in New version is called load_types_definition(). Concerning lxml if it does not work well on Windows we should open a bug request and discuss what xml parser to use.

[oroulet]

Guy checkout typedef if you use git

[mpf82]

@oroulet Sorry, my bad, I did indeed check out the current master.

However, testing with the typedef branch yields the same result:

Traceback (most recent call last):
  File "...\testing\opc_ua_array_tests.py", line 64, in <module>
    client.load_type_definitions(nodes=None)
  File "...\opcua\client\client.py", line 579, in load_type_definitions
    d = generator.get_python_classes(structs_dict)
  File "...\opcua\common\structures_generator.py", line 206, in get_python_classes
    exec(code, env)
  File "<string>", line 4
    class "Bockdata_1"."Bockdaten"(object):

[oroulet]

Ok thanks, can you attach the xml definition to this bug?

[mpf82]

Sure. I've removed quite a bit, but this should be enought to reproduce the issue:

<?xml version="1.0" encoding="utf-8"?>
<opc:TypeDictionary DefaultByteOrder="LittleEndian"
TargetNamespace="http://www.siemens.com/simatic-s7-opcua"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:opc="http://opcfoundation.org/BinarySchema/" xmlns:ua="http://opcfoundation.org/UA/"
xmlns:tns="http://www.siemens.com/simatic-s7-opcua">
  <opc:Import Namespace="http://opcfoundation.org/UA/" />
  <opc:StructuredType BaseType="ua:ExtensionObject"
  Name="&quot;Bockdata_1&quot;.&quot;Bockdaten&quot;">
    <opc:Field TypeName="opc:Int32" Name="JobData_Size" />
    <opc:Field TypeName="tns:JobData_ATM_Test" Name="JobData" LengthField="JobData_Size" />
    <opc:Field TypeName="opc:Int32" Name="ResultData_Size" />
    <opc:Field TypeName="tns:Data_backflash_ATM_1" Name="ResultData"
    LengthField="ResultData_Size" />
  </opc:StructuredType>
  <opc:StructuredType BaseType="ua:ExtensionObject" Name="Bockdata_1">
    <opc:Field TypeName="tns:&quot;Bockdata_1&quot;.&quot;Bockdaten&quot;" Name="Bockdaten" />
  </opc:StructuredType>
  <opc:StructuredType BaseType="ua:ExtensionObject"
  Name="&quot;Bockdata&quot;.&quot;Bockdaten&quot;">
    <opc:Field TypeName="opc:Int32" Name="JobData_Size" />
    <opc:Field TypeName="tns:JobData_ATM_Test" Name="JobData" LengthField="JobData_Size" />
    <opc:Field TypeName="opc:Int32" Name="ResultData_Size" />
    <opc:Field TypeName="tns:Data_backflash_ATM" Name="ResultData" LengthField="ResultData_Size" />
  </opc:StructuredType>
  <opc:StructuredType BaseType="ua:ExtensionObject" Name="Bockdata">
    <opc:Field TypeName="tns:&quot;Bockdata&quot;.&quot;Bockdaten&quot;" Name="Bockdaten" />
  </opc:StructuredType>
  <opc:StructuredType BaseType="ua:ExtensionObject" Name="Data_backflash_ATM">
    <opc:Field TypeName="opc:Float" Name="value_A_01" />
    <opc:Field TypeName="opc:Float" Name="value_A_02" />
    <opc:Field TypeName="opc:Float" Name="value_B_01" />
    <opc:Field TypeName="opc:Float" Name="value_B_02" />
    <opc:Field TypeName="opc:String" Name="value_string" />
    <opc:Field TypeName="opc:Int16" Name="StatusA" />
    <opc:Field TypeName="opc:Int16" Name="Unit" />
    <opc:Field TypeName="opc:Int16" Name="Attemtps" />
    <opc:Field TypeName="opc:Int16" Name="job_type" />
    <opc:Field TypeName="opc:Int16" Name="Process_value_no" />
    <opc:Field TypeName="opc:Int16" Name="StatusB" />
  </opc:StructuredType>
  <opc:StructuredType BaseType="ua:ExtensionObject"
  Name="&quot;JobControl_ATM&quot;.&quot;job_control_bits&quot;">
    <opc:Field TypeName="opc:Boolean" Name="Bit1" />
    <opc:Field TypeName="opc:Boolean" Name="Bit2" />
    <opc:Field TypeName="opc:Boolean" Name="Bit3" />
    <opc:Field TypeName="opc:Boolean" Name="Bit4" />
    <opc:Field TypeName="opc:Boolean" Name="Bit5" />
    <opc:Field TypeName="opc:Boolean" Name="Bit6" />
    <opc:Field TypeName="opc:Boolean" Name="Bit7" />
    <opc:Field TypeName="opc:Boolean" Name="Bit8" />
  </opc:StructuredType>
  <opc:StructuredType BaseType="ua:ExtensionObject" Name="Anwenderdatentyp_1">
    <opc:Field TypeName="opc:String" Name="12" />
  </opc:StructuredType>
</opc:TypeDictionary>

If you need the complete XML, I can upload it to pastebin.

Let me also suggest this (partial) fix: https://pastebin.com/iGDJ2RDA (raw) (expires in 1 week)

This should also handle numeric variables, such as <opc:Field TypeName="opc:String" Name="12" /> - though I'm not sure how registering will work then.

With the code from my pastebin I now can get past get_python_classes(), but a new exception is raised:

Traceback (most recent call last):
  File "...\testing\opc_ua_array_tests.py", line 64, in <module>
    client.load_type_definitions(nodes=None)
  File "...\opcua\client\client.py", line 655, in load_type_definitions
    ua.register_extension_object(name, nodeid, structs_dict[name])
KeyError: 'SimaticStructures'

Ofc you do not have to use my edited code from the pastebin.

Cheers =)

[mpf82]

I also changed client.py as follows:

from opcua.common.structures_generator import StructGenerator, _clean_name

...

            # register classes
            # every children of our node should represent a class
            print("structs_dict", structs_dict.keys())
            for ndesc in node.get_children_descriptions():
                ndesc_node = self.get_node(ndesc.NodeId)
                ref_desc_list = ndesc_node.get_references(refs=ua.ObjectIds.HasDescription, direction=ua.BrowseDirection.Inverse)
                if ref_desc_list:  #some server put extra things here
                    name = _clean_name( ndesc.BrowseName.Name )
                    nodeid = ref_desc_list[0].NodeId
                    try:
                        ua.register_extension_object(name, nodeid, structs_dict[name])
                    except Exception as e:
                        print("ndesc.BrowseName.Name", name)
                        print("ERROR:", e)

Output:

structs_dict dict_keys(['ua', 'Nem_DataContainer', '_Nem_Orderheader_ordertype_', 'Nemetris_Traceabilitydata_NEM', 'Nem_JobInformation_OverviewJobsPerOP', 'JobControl_ATM', '_EOLCheck_1_eol_plc_', 'Nem_StationInfo', 'FetchResultDataDataType', '_OptionControl_optioncontrol_plc_', 'uuid', 'Nemetris_Traceability_Preset_PLC', '_OptionControl_1_plc_optioncontrol_', 'FetchResultErrorDataType', '__builtins__', 'NEM_Telegram_6', '_JobData_ATM_Test_job_control_bits_', 'FetchResultDataType', 'Nem_ComponentVariant_Control_Bits', 'Nemetris_Traceability_Preset_NEM', 'NEM_Telegram_14', 'NEM_Telegram_5', '_EOLCheck_1_plc_eol_', 'Data_backflash_ATM_1', 'Nem_Header', '_Bockdata_1_Bockdaten_', 'NEM_Telegram_1', 'EOLCheck', 'NEM_Telegram_3', 'datetime', '_Nem_JobInformation_JobInfos_job_control_bits_', 'Header', 'Data_backflash_ATM', 'Nemetris_Orderdata_PLC', 'OptionControl', 'ParameterResultDataType', '_JobControl_job_control_bits_', 'Optioncode', 'Bockdata', 'NEM_Telegram_100', '_Bockdata_Bockdaten_', 'Bockdata_1', '_OptionControl_1_optioncontrol_plc_', '_OptionControl_plc_optioncontrol_', 'JobDatabackflash', '_Nem_StationInfo_work_instruction_', 'Nem_Orderheader', 'JobData_ATM_Test', '_Nem_StationInfo_op_status_', '_Header_errorcode_', '_EOLCheck_plc_eol_', 'Optioncode_1', 'Nemetris_Orderdata_NEM', '_JobControl_ATM_job_control_bits_', 'Nem_ComponentVariant', 'Nemetris_JobMasterlist_PLC', 'EOLCheck_1', 'Nemetris_Traceabilitydata_PLC', '_JobDatabackflash_job_control_bits_', 'OptionControl_1', 'NEM_Telegram_4', '_EOLCheck_eol_plc_', 'JobStatus', 'Nem_JobInformation_JobInfos', '_PreOrder_job_control_bits_', 'NEM_Telegram_7', 'Nemetris_JobMasterlist_NEM', 'Anwenderdatentyp_1', 'JobControl', '_Nem_StationInfo_control_bits_', '_Nem_Header_errorcode_'])
ndesc.BrowseName.Name SimaticStructures
ERROR: 'SimaticStructures'
ndesc.BrowseName.Name SimaticSystemStructures
ERROR: 'SimaticSystemStructures'

As you can see, there are no structures called SimaticStructures and SimaticSystemStructures in the dictionary (and I can't figure out where they are coming from), but all other extension objects seem to register fine. Example: registring new extension object: _Bockdata_1_Bockdaten_ StringNodeId(ns=3;s=TE_"Bockdata_1"."Bockdaten") <class '_Bockdata_1_Bockdaten_'>

[oroulet]

@mpf82 To register the new structures we are simply getting all the children of the SimaticStructures node. so if we look for it then it must be a children....

Otherwise, just removing " and . from names as you do should fix the stuff. Maybe I should move that method to the structure_generator.py file so it can be used on server side too. there is nothing client specific there

[oroulet]

@mpf82 I added one more commit with you proposition. Can you update and test to see I did not forget something? (obs you might need to reclone/reset since I had to force a commit push

[mpf82]

@oroulet Thanks for your edits. Cleaning the name needs also be done in _make_from_binary() and _make_model()

Here's my edited structures.py: https://pastebin.com/raw/6eqUGz87

Please note that I changed the imports as follows:

#===================================================================================================
# Support either LXML or xmltodict as XML parser
#===================================================================================================
OPCUA_XML_PARSER = None
OPCUA_XML_PARSER_LXML = "LXML"
OPCUA_XML_PARSER_XMLTODICT = "XMLTODICT"
try:
    from lxml import objectify
    OPCUA_XML_PARSER = OPCUA_XML_PARSER_LXML
except ImportError:
    import xmltodict
    OPCUA_XML_PARSER = OPCUA_XML_PARSER_XMLTODICT

Now we could both work on the same version of the file w/o me always editing the lxml part to make it run on our system.

Btw, will print("Error {} is found as child of binary definition node but is not found in xml".format(name)) not break the Python 2.x compatibility (I'm not sure, we're only using 3.4+)? Maybe use the logger from uatypes instead?

[mpf82]

I just noticed: your version of _cleanname() completely replaces numerical variables with ``while my previous version does prefix them with_``.

I think the prefix version is better, otherwise, if you have multiple numerical variables, they will all be replaced by _.

I'm not a friend of numerical variables, but it's part of the Siemens structure we have to work with.

For comparison:

def _clean_name(name):
    """
    Remove characters that might be present in  OPC UA structures
    but cannot be part of of Python class names
    """
    name = re.sub(r'\W+', '_', name)
    name = re.sub(r'^[0-9]+', '_', name)
    return name

def _clean_name2(name):
    _name = re.sub(r'\W+', '_', name)
    # test for numerical variables
    try:
        int(_name)
        return "_{}".format( _name )
    except:
        pass
    return _name

print(_clean_name("12")) # Output: _
print(_clean_name2("12")) # Output: _12

[oroulet]

@mpf82 thanks for the feedback.Can you send me the diff instead of complete code? That is easier to see what you changed, so I do not miss half of it ;-)

[mpf82]

@oroulet Here's the Patch/Diff (generated with WinMerge, Style: Unified): https://pastebin.com/JqZbnNkv

Though I think you should be able to simply use the file, w/o merging it, because I made sure to support both LXML and xmltodict.

Note: The diff includes my changes to _clean_name(), while my previous pastebin still uses your version.

[oroulet]

I pushed one more attempt at fixing it ;-) unfortunately I have no server to test against. so you need to do the testing....

[oroulet]

why can't you install lxml? isnt't avaialbel with pip?

[mpf82]

It's available in pip, however building fails in MS Visual Studio because it can not find libxml2 and I didn't get it to work by downloading the pre-built DLLs.

We're already using xmltodict in productive environments - it's just one pure-python file that does the job, that's why I changed the code instead of investing more time in trying to get lxml to run.

And shipping a single python file to a customer instead of shipping a huge library that has to be specifically built for the target system is always easier. :)

[mpf82]

Your changes are looking good, thumbs up :)

Still, I'd like you to reconsider the try..except import block to support both lxml and xmltodict.

My adapted structures.py file: https://pastebin.com/3MhmMVFc Diff: https://pastebin.com/KGKm3vEr

Cheers =)

[oroulet]

OK thanks, The xml stuff is something completely different. So far I have not been convinced by any xml libraries I have seen. We just try to survive the xml stuff......

[zerox1212]

I have not had issues with lxml on windows. As far as I know lxml is the fastest xml library for python.

I am not sure why we changed to lxml, most of the xml stuff worked with stdlib etree in earlier versions.

[oroulet]

I think we changed to lxml because you advised it ;-)

On Tue, Sep 12, 2017, 17:23 Andrew notifications@github.com wrote:

I have not had issues with lxml on windows. As far as I know lxml is the fastest xml library for python.

I am not sure why we changed to lxml, most of the xml stuff worked with stdlib etree in earlier versions.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/FreeOpcUa/python-opcua/issues/474#issuecomment-328876817, or mute the thread https://github.com/notifications/unsubscribe-auth/ACcfzlL-BpY8nlbeeemA1pbkhjbeq90Sks5shpuAgaJpZM4Octxx .

[zerox1212]

It wasn't me. If you look at the XML stuff I was involved with in common (importer, parser, exporter) it uses Etree.

Looks like only the extension object stuff uses lxml. from lxml import objectify

Objectify and Etree gernally should not be mixed in a single module to avoid confusion. However I'm sure objectify is worth the import for the extension objects.

[the-digital-engineer]

With the lxml issue: what about a more fundamental solution. The inclusion of packages is Python's weakest aspect. Please consider pipenv. You may declare pip packages straight forward.

Yeah, with Windows it's sometimes hard when pip packages try to compile something themselves. In 98% of those cases, gohlke ( https://www.lfd.uci.edu/~gohlke/pythonlibs/ ) may help you within seconds.

My approach since some months is as follows: pip packages where pip works. Full URL links to gohlke packages where necessary. Never looked back.

[darxridge]

Thank you guys for python opcua!

Here is the minimal code which I use to read and print a variable from a Siemens CPU. Note the quoting in NodeId. I hope, it will be helpful for someone:

from opcua import Client
from opcua import ua
if __name__ == "__main__":
    client = Client("opc.tcp://192.168.26.11:4840")
    try:
        client.connect()
        print(client.get_node(ua.NodeId('"IDB_Data"."AA"."Value_1"', 3)).get_value())
    finally:
        client.disconnect()

[driftregion]

I'm facing the same issue (Also on a Siemens OPC UA server).

From the above discussion, it looks like the both the utf-8 XML decode issue and the code generation issue (dealing with the "." in Siemens' naming) remain in current master 13b4e1249e06a3d3feef09afd04139eb774207bc

I've worked around both with these changes https://github.com/FreeOpcUa/python-opcua/compare/master...driftregion:bugfix/474

They don't seem to break any tests

----------------------------------------------------------------------
Ran 407 tests in 80.542s

OK

I've not yet tested the generated code to see if the name mangling has broken something, I'll report back with what I find.

[driftregion]

Looks OK, I've opened https://github.com/FreeOpcUa/python-opcua/pull/1081 and added a test for code generation with Siemens-like struct names.