Issue in IEC61850 adapter and simulator.

I am new to OSGP. I have done the installation with Manual set. There were no errors in installation. After setting up the platform as per https://documentation.gxf.lfenergy.org/Userguide/Installation/setupOSGP.html , we started the protocol-simulator-iec61850 which runs successfully and started publishing events with ServerSap object. But osgp-protocol-adapter-iec61850 is not showing any logs of receiving the publish data from simulator. There are no WSDL files / SOAP req for IEC61850 device registration/addition. What is the process to add/register the protocol-simulator-iec61850 devices on platform and get there dynamic attribute values from application layer via SOAP Request? we want to have the list of all iec61850 devices present and get current state readings.

Hello @Rupali-2,

the short answer to your questions is: indeed you need to create some IEC61850 device(s) in the platform databases and make sure the device(s) connect to the simulator application. When the connection is made, buffered or unbuffered reporting will be enabled and then the published reports can be received by the platform.

The longer and more complicated answer is as follows.

The standard configuration for the IEC61850 simulator has been set up for a microgrids RTU. The microgirds RTU can't be created using SOAP calls, only using SQL scripts. We are no longer using this type of smart device for now. There is a test-suite that uses this functionality, but not the simulator, see the project cucumber-tests-platform-microgrids.
The IEC61850 simulator can be configured as a light measurement device. See the server model here: https://documentation.opensmartgridplatform.org/Protocols/IEC61850/FlexOVL_540_171101_2/FlexOVL_540_171101_2_out.icd.html
The IEC61850 simulator can be configured as a light switching device, well a very basic version of such a device. See the server model here: https://documentation.opensmartgridplatform.org/Protocols/IEC61850/SWDevice-010805/SWDevice-010805.icd.html

For the light switching device, SOAP calls can be used to add an SSLD(s) using the IEC61850 protocol. For the light measurement device, we are using SQL scripts to create some devices using the IEC61850 protocol. I see you found this issue: https://github.com/OSGP/Documentation/issues/236. It links to some example SQL scripts to create light measurement devices. And a pull request has been opened to add some SOAP operations that can create/modify and read light measurement devices.

Note, these different simulator configurations are mutually exclusive, so running all 3 of the mentioned device types at once is not possible using 1 simulator application instance.

Which of the IEC61850 device types are you interested in?

Thanks for your inputs. We are interested in RTU and energy/gas meters.
however as suggested by you we are currently testing the LMD device specified in 2nd point.(i.e https://documentation.opensmartgridplatform.org/Protocols/IEC61850/FlexOVL_540_171101_2/FlexOVL_540_171101_2_out.icd.html). We checked out development branch and we came across following error while calling GetStatusRequest request.

Log of protocol adaptor: `2020-11-24 16:35:38.731] [ip-172-31-26-161] [protocolIec61850InboundOsgpCoreRequestsMessageListenerContainer-25] INFO o.o.a.p.i.i.n.s.Iec61850DeviceConnectionService@testIfConnectionIsCachedAndAlive:206 - Trying to find connection in cache for deviceIdentification: LMD01 2020-11-24 16:35:38.731] [ip-172-31-26-161] [protocolIec61850InboundOsgpCoreRequestsMessageListenerContainer-25] INFO o.o.a.p.i.i.n.s.Iec61850DeviceConnectionService@testIfConnectionIsCachedAndAlive:210 - Connection found for deviceIdentification: LMD01 2020-11-24 16:35:38.731] [ip-172-31-26-161] [protocolIec61850InboundOsgpCoreRequestsMessageListenerContainer-25] INFO o.o.a.p.i.i.n.s.Iec61850DeviceConnectionService@testIfConnectionIsCachedAndAlive:215 - Testing if connection is alive using AA1TH01LD0/LLN0.NamPlt for deviceIdentification: LMD01 2020-11-24 16:35:38.731] [ip-172-31-26-161] [protocolIec61850InboundOsgpCoreRequestsMessageListenerContainer-25] ERROR o.o.a.p.i.i.n.Iec61850Client@readNodeDataValues:235 - IOException during readNodeDataValues java.io.IOException: Error sending packet. at com.beanit.openiec61850.ClientAssociation.encodeWriteReadDecode(ClientAssociation.java:418) at com.beanit.openiec61850.ClientAssociation.getDataValues(ClientAssociation.java:795) at org.opensmartgridplatform.adapter.protocol.iec61850.infra.networking.Iec61850Client.readNodeDataValues(Iec61850Client.java:230) at org.opensmartgridplatform.adapter.protocol.iec61850.infra.networking.services.Iec61850DeviceConnectionService.testIfConnectionIsCachedAndAlive(Iec61850DeviceConnectionService.java:220) at org.opensmartgridplatform.adapter.protocol.iec61850.infra.networking.services.Iec61850DeviceConnectionService.connect(Iec61850DeviceConnectionService.java:106) at org.opensmartgridplatform.adapter.protocol.iec61850.infra.networking.services.Iec61850DeviceConnectionService.connect(Iec61850DeviceConnectionService.java:94) at org.opensmartgridplatform.adapter.protocol.iec61850.infra.networking.services.Iec61850LmdDeviceService.connectToDevice(Iec61850LmdDeviceService.java:101) at org.opensmartgridplatform.adapter.protocol.iec61850.infra.networking.services.Iec61850LmdDeviceService.getStatus(Iec61850LmdDeviceService.java:63) at org.opensmartgridplatform.adapter.protocol.iec61850.infra.messaging.processors.PublicLightingGetLightSensorStatusRequestMessageProcessor.processMessage(PublicLightingGetLightSensorStatusRequestMessageProcessor.java:89) at org.opensmartgridplatform.adapter.protocol.iec61850.infra.messaging.DeviceRequestMessageListener.onMessage(DeviceRequestMessageListener.java:60)

At this moment, we are not using any energy or gas meters which implement the IEC61850 protocol. Have a look at the DLMS protocol adapter if you are interested in those types of smart devices.

Regarding the error you have posted here, I think you might be running into a configuration problem with the protocol adapter and simulator. It is interesting to see that there seems to be a connection, as an active connection is fetched from the connection cache.

Configuration for the osgp-adapter-protocol-iec61850 component (osgp-adapter-protocol-iec61850.properties):

#Buffered reporting is used for production, the default value is true.
#Unbuffered reporting is used for development and testing, set value to false.
iec61850.is.buffered.reporting.enabled=false

#Folder containing ICD files for IEC61850 devices that have a file name configured in
#the protocol database (see the table iec61850_device).
iec61850.icd.files.folder=/path/to/folder/containing/the/server/model/file/

Buffered reporting needs to be changed to unbuffered reporting. This is due to an implementation limitation in the OpenMUC IEC61850 library which is used by the protocol adapter and the simulator. The folder as configured needs to contain the server model / ICD file (FlexOVL_540_171101_2_out.icd) and this file needs to be readable by the tomcat user.

Configuration for the osgp-protocol-simulator-iec61850 component (osgp-simulator-protocol-iec61850.properties):

#Port for the IEC61850 protocol. Default is 102. If there are problems using a port below 1024, consider using a different 
#port for the simulator, and update the database records for the protocol adapter accordingly.
rtu.port=102

#The path to the server model / ICD file.
rtu.icd=/path/to/folder/containing/the/server/model/file/FlexOVL_540_171101_2_out.icd

#The name of the IED / smart device.
rtu.serverName=AA1TH01

#Disable generating reports for the microgrids implementation.
rtu.stopGeneratingValues=true

Change the line:

<ReportControl name="rcb_A" rptID="A" datSet="StatNrmlA" confRev="1" bufTime="500" buffered="true">

<ReportControl name="rcb_A" rptID="A" datSet="StatNrmlA" confRev="1" bufTime="500">

for the server model / ICD file FlexOVL_540_171101_2_out.icd. This will configure unbuffered reporting instead of buffered reporting in the server model.

I think these configuration options should get you a step further along. In case you encounter errors, please post the full logging of the protocol adapter component, preferably your first connection attempt, i.e. no connection is present in the connection cache yet.

As per your suggestion, we are looking at DLMS protocol adapter for energy meters. But to test this, can you provide the DLMS simulator. We also came across this jira issue https://smartsocietyservices.atlassian.net/browse/OC-682 on Moving DLMS simulator from Closed to Open Source repository. As DLMS protocol adapter handles energy meters, what are the SOAP API's present in platform for DLMS energy devices. ( Like to get meter readings etc.).

For MQTT protocol, currently osgp-adapter-domain-distributionautomation will detect the RTU device in the database and send a GET_DATA request to establish communication. Is this response of GET_DATA gets persisted anywhere?

I'm sorry, I can't move the DLMS simulator project to the Open Source repository. The project management will have to make that decision. I added a little link to this GitHub issue in the Jira ticket...

The SOAP web services for smart meter devices are part of the osgp-adapter-ws-smartmetering component. And I think a couple of more generic SOAP operations are used from the osgp-adapter-ws-core component. Here are the WSDL/XSD files for the smart meters: https://github.com/OSGP/open-smart-grid-platform/tree/development/osgp/shared/osgp-ws-smartmetering/src/main/resources

When the osgp-adapter-domain-distributionautomation component sends a GET_DATA request, the response for this request can be forwarded to the osgp-adapter-ws-distributionautomation component. This is controlled by the integration_type column of the device table of the osgp_core database. When this column is set to WEB_SERVICE, the response will be forwarded to the web service component and persisted in the response_data table of the osgp_adapter_ws_distributionautomation database. The data of the message is saved to the message_data column, along with some metadata in some other columns. Alternatively, if the integration_type column is set to KAFKA, the response can be published to a Kafka topic.

What are the generic SOAP operations are used from the osgp-adapter-ws-core component for MQTT devices? we tried osgp-adapter-ws-core soap end points like getStatus,getconfiguration but got following response. kindly let us know if we missing out on something.

2020-11-27 18:28:56.756] [ip-172-31-26-161] [DefaultMessageListenerContainer-1] INFO o.o.c.i.j.p.o.ProtocolRequestMessageSender@send:51 - Sending protocol request message for device [TST-01] using protocol [MQTT] with version [3.1.1]
2020-11-27 18:28:56.756] [ip-172-31-26-161] [DefaultMessageListenerContainer-1] INFO o.o.c.i.j.p.o.ProtocolRequestMessageSender@send:56 - Message sender destination queue: [queue://protocol-mqtt.1_0.osgp-core.1_0.requests] for protocol [MQTT] with version [3.1.1]
2020-11-27 18:28:56.757] [ip-172-31-26-161] [DefaultMessageListenerContainer-1] INFO o.o.c.i.j.p.o.ProtocolRequestMessageSender@sendMessage:64 - Sending request message to protocol requests queue
2020-11-27 18:28:56.774] [ip-172-31-26-161] [ActiveMQ Task-1] INFO o.a.a.t.failover.FailoverTransport@doReconnect:1054 - Successfully connected to tcp://localhost:61616
2020-11-27 18:28:56.801] [ip-172-31-26-161] [DefaultMessageListenerContainer-1] INFO o.o.c.i.j.d.i.DomainRequestMessageListener@onMessage:59 - Domain request for device [TST-01] processed.
2020-11-27 18:28:56.850] [ip-172-31-26-161] [protocolMqttInboundOsgpCoreRequestsMessageListenerContainer-1] INFO o.o.a.p.m.a.c.m.InboundOsgpCoreRequestMessageListener@onMessage:37 - Received message of type: GET_STATUS
2020-11-27 18:28:56.851] [ip-172-31-26-161] [protocolMqttInboundOsgpCoreRequestsMessageListenerContainer-1] ERROR o.o.s.i.jms.BaseMessageProcessorMap@getMessageProcessor:56 - No message processor found in InboundOsgpCoreRequestsMessageProcessorMap for message type: GET_STATUS
2020-11-27 18:28:56.854] [ip-172-31-26-161] [protocolMqttInboundOsgpCoreRequestsMessageListenerContainer-1] ERROR o.o.a.p.m.a.c.m.InboundOsgpCoreRequestMessageListener@onMessage:48 - Exception: Message processor not configured 
javax.jms.JMSException: Message processor not configured
    at org.opensmartgridplatform.shared.infra.jms.BaseMessageProcessorMap.getMessageProcessor(BaseMessageProcessorMap.java:58)
    at org.opensmartgridplatform.shared.infra.jms.BaseMessageProcessorMap.getMessageProcessor(BaseMessageProcessorMap.java:50)
    at org.opensmartgridplatform.adapter.protocol.mqtt.application.config.messaging.InboundOsgpCoreRequestMessageListener.onMessage(InboundOsgpCoreRequestMessageListener.java:42)

Also, osgp-adapter-domain-distributionautomation does't sends a GET_DATA request after starting ( version - 5.6.0)and logs show following data

2020-11-27 18:13:32.380] [ip-172-31-26-161] [ActiveMQ Task-1] INFO o.a.a.t.failover.FailoverTransport@doReconnect:1054 - Successfully connected to tcp://localhost:61616
2020-11-27 18:13:32.415] [ip-172-31-26-161] [main] INFO o.s.web.context.ContextLoader@initWebApplicationContext:307 - Root WebApplicationContext initialized in 7002 ms
2020-11-27 18:14:00.000] [ip-172-31-26-161] [da-communication-monitoring-1] INFO o.o.a.d.d.a.t.CommunicationMonitoringTask@run:59 - Running communication monitoring task.
2020-11-27 18:14:00.409] [ip-172-31-26-161] [da-communication-monitoring-1] INFO o.o.a.d.d.a.t.CommunicationMonitoringTask@run:64 - Communication Monitoring Task already running. Skipping this run.
2020-11-27 18:15:00.001] [ip-172-31-26-161] [da-communication-monitoring-1] INFO o.o.a.d.d.a.t.CommunicationMonitoringTask@run:59 - Running communication monitoring task.
2020-11-27 18:15:00.016] [ip-172-31-26-161] [da-communication-monitoring-1] INFO o.o.a.d.d.a.t.CommunicationMonitoringTask@run:64 - Communication Monitoring Task already running. Skipping this run.
2020-11-27 18:16:00.000] [ip-172-31-26-161] [da-communication-monitoring-1] INFO o.o.a.d.d.a.t.CommunicationMonitoringTask@run:59 - Running communication monitoring task.
2020-11-27 18:16:00.015] [ip-172-31-26-161] [da-communication-monitoring-1] INFO o.o.a.d.d.a.t.CommunicationMonitoringTask@run:64 - Communication Monitoring Task already running. Skipping this run.

I don't think any of the SOAP operations that are part of the osgp-adapter-ws-core component are mapped to/implemented in the osgp-adapter-protocol-mqtt component.

The communication monitoring task is in state RUNNING. Either delete the record(s) in table task of database osgp_core, or update the column task_status to state AVAILABLE.

SSL keystore pfx file

SOAP API require organisation's .pfx file for authentication. If we want to build and UI Application client where users from various organisation login and make soap request on behalf of organisation. Is it required to store .pfx file (crypto material of each organisation) on user's machine? if not what are the approach's for .pfx file management for server?

FIND Devices SOAP API Filter Issue

In DeviceManagement.wsdl we have FindDevicesRequest SOAP request. In body template in all tags are marked as optional But if we apply only filter in request body we get validation error. It says other tag details are also required. Does FIND Devices SOAP API not support the filtering on only few tags?

<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
   <SOAP-ENV:Header/>
   <SOAP-ENV:Body>
      <SOAP-ENV:Fault>
         <faultcode>SOAP-ENV:Client</faultcode>
         <faultstring xml:lang="en">Validation error</faultstring>
         <detail>
            <spring-ws:ValidationError xmlns:spring-ws="http://springframework.org/spring-ws">cvc-complex-type.2.4.b: The content of element 'ns1:DeviceFilter' is not complete. One of '{"http://www.opensmartgridplatform.org/schemas/devicemanagement/2014/10":DeviceIdentification, "http://www.opensmartgridplatform.org/schemas/devicemanagement/2014/10":Alias, "http://www.opensmartgridplatform.org/schemas/devicemanagement/2014/10":City, "http://www.opensmartgridplatform.org/schemas/devicemanagement/2014/10":PostalCode, "http://www.opensmartgridplatform.org/schemas/devicemanagement/2014/10":Street, "http://www.opensmartgridplatform.org/schemas/devicemanagement/2014/10":Number, "http://www.opensmartgridplatform.org/schemas/devicemanagement/2014/10":Municipality, "http://www.opensmartgridplatform.org/schemas/devicemanagement/2014/10":DeviceType, "http://www.opensmartgridplatform.org/schemas/devicemanagement/2014/10":Manufacturer, "http://www.opensmartgridplatform.org/schemas/devicemanagement/2014/10":Model, "http://www.opensmartgridplatform.org/schemas/devicemanagement/2014/10":DeviceExternalManaged, "http://www.opensmartgridplatform.org/schemas/devicemanagement/2014/10":DeviceActivated, "http://www.opensmartgridplatform.org/schemas/devicemanagement/2014/10":DeviceInMaintenance, "http://www.opensmartgridplatform.org/schemas/devicemanagement/2014/10":SortDir, "http://www.opensmartgridplatform.org/schemas/devicemanagement/2014/10":SortedBy, "http://www.opensmartgridplatform.org/schemas/devicemanagement/2014/10":HasTechnicalInstallation}' is expected.</spring-ws:ValidationError>
         </detail>
      </SOAP-ENV:Fault>
   </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

SOAP Request Passed:

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns="http://www.opensmartgridplatform.org/schemas/common/2014/10" xmlns:ns1="http://www.opensmartgridplatform.org/schemas/devicemanagement/2014/10">
   <soapenv:Header>
      <ns:ApplicationName>soapUI</ns:ApplicationName>
      <ns:UserName>rupali</ns:UserName>
      <ns:OrganisationIdentification>test-org</ns:OrganisationIdentification>
   </soapenv:Header>
   <soapenv:Body>
      <ns1:FindDevicesRequest>
         <!--Optional:-->
         <ns1:DeviceFilter>
            <!--Optional:-->
            <!--type: Identification-->
            <ns1:OrganisationIdentification>test-org</ns1:OrganisationIdentification>
         </ns1:DeviceFilter>
      </ns1:FindDevicesRequest>
   </soapenv:Body>
</soapenv:Envelope>

OSGP / Documentation

Issue in IEC61850 adapter and simulator. #238

SSL keystore pfx file

FIND Devices SOAP API Filter Issue