Reconnection issue after server restarts - missing event

anthrogan commented 5 years ago

I'm submitting a ... (check one with "x") Happy to submit pull request

Problem

(x) bug report => please search GitHub for a similar issue or PR before submitting
( ) unexpected behaviour => please visit the Wiki before

Information

Current behaviour

When a server running a OPC server is restarted OPC server does not shutdown in a normal fashion, meaning Node-OPCUA emits 'connection_lost' event. iiot-opcua has no catch for this event so the session remains open and never closes, on reconnection of server it try's to open another session and receives

ERROR !!! , please check here !!!! callback may be called twice !! Error

Expected behaviour

Disconnect properly when OPC server is shutdown/restarted.

Minimal reproduction of the problem with instructions

Host OPC server on different machine
Connect some tags to it via node-red
Restart server

Your Environment

Please tell us about your environment:

(x) I am using just the Node-RED package and got node-opcua installed with it
( ) I have installed node-opcua global from source (using git clone)
( ) I have installed node-opcua as a global package (using npm i -g)
Device :
- ( ) macOS: v
- ( ) Linux: v
- ( ) Windows: v
- ( ) Raspbian: v
- (x) Other:
  - Name: Docker
  - Version:
- Description of the connecting OPC UA system:
  - Name : Kepware
  - Version: 5.9
  - Vendor :
  - Link : https://

node-red-contrib-iiot-opcua version: ``
Node: (min. 8.x) node --version =

biancode commented 5 years ago

maybe that is fixed with v0.5 of node-opcua - node-opcua want to handle that case itself like the uaexpert - would you send a pull request or just report?

biancode commented 5 years ago

Are you looking into the 0.4.6 source of node-opcua or 0.5.+? Like the uaexpert the connection should stay in open mode and the node-opcua should do the reconnect. If the connection is lost the state machine of the iiot-opcua connector node should handle the reconnecting process. That works very well with simulators and hardware on my side.

anthrogan commented 5 years ago

iiot-opcua does not know connection_lost has been emitted, but once it gets re-connection it tries to overwrite itself which throws the node-opcua error i quoted (I think this is whats happening)

In a node-opcua service I have, I recreate the session on connection_reestablished but node-opcua takes care of the re-connection initially. I'll do some more looking when I get back to work tomorrow and I'll try and post some better examples with pictures 😃 thanks for the quick response though

biancode commented 5 years ago

With some more details I could get a better understanding of your problem. How do you restart? Via Node-RED deploy or restart from CMD node?

anthrogan commented 5 years ago

Hi sorry for late reply!

First of all I apologise I described the wrong event, I think IIoT needs to recreate the session on connection_reestablished not connection_lost! I'm terrible at explaining my issues!

Edit: I experience this error only when force resetting a remote OPC Server!

In a Node-OPCUA NodeJS application that I have, I recreate the session on connection_reestablished so the subscription has something to work with:

    opc.on("connection_lost", () => {
      opcClient.publish('powertrain/unit/status', "[OPCUA Server] - connection lost - attempting to reconnect");
    })
    opc.on("connection_reestablished", () => {
      opc.createSession((err, session) => {
        if (!err) {
          theSession = session;
          opcClient.publish('powertrain/unit/status', "[OPCUA Server] - session has been created");
          subSetup();
        } else {
          opcClient.publish('powertrain/unit/status', "[OPCUA Server] - session cannot be created");
        }
      })
    });

Without recreating the session on connection_reestablished I found that the session didn't seem to recreate on its own and instead I would see node-opcua error being thrown:

ERROR !!! , please check here !!!! callback may be called twice !! Error

My understanding is that its trying to create subscriptions without/before a session is created. I appear to be getting the same error in IIoT whenever we have a server reset that is running the OPC server (Unfortunately this is production server so I'm not able to reset randomly at the minute!) and I think it is because IIoT does not attempt to recreate the session once connection is reestablished with the OPC Server - Example with createSession added:

  node.bianco.iiot.opcuaClient.on('connection_reestablished', function () {
    connectorLib.internalDebugLog('!!!!!!!!!!!!!!!!!!!!!!!!  CLIENT CONNECTION RE-ESTABLISHED !!!!!!!!!!!!!!!!!!!'.bgWhite.orange)
    connectorLib.internalDebugLog('CONNECTION RE-ESTABLISHED: ' + node.endpoint)
    node.bianco.iiot.stateMachine.unlock()
   // I think we need to recreate the session here
  node.bianco.iiot.opcuaClient.createSession(node.bianco.iiot.userIdentity || {})
  })

So yeah without a session, a subscription can't monitor items. This is at least how I understand I have added some code to the bottom how I work with session/subscription in my Node application so it hopefully makes a bit more sense and how I am approaching the issue :smile: I know a lot of this is Node-OPCUA related but I think there it is only a small change missing from IIoT.

Further example of how I interact with session/subscription,


  // subscription settings
  theSubscription = new opcua.ClientSubscription(theSession, {
    requestedPublishingInterval: 1000,
    requestedLifetimeCount: 10 * 60 * 10,
    requestedMaxKeepAliveCount: 10,
    maxNotificationsPerPublish: 200,
    publishingEnabled: true,
    priority: 10
  });

  // Import all the tags from tag file
  let items = require("../../config/opcTags/tags");
  // for each tag in the tag file, setup a subscription
  items.forEach((item) => {
    let nodeId = "ns=2;s=" + item;
    let monitoredItem = theSubscription.monitor({
      nodeId: opcua.resolveNodeId(nodeId),
      attributeId: opcua.AttributeIds.Value
    }, {
        samplingInterval: 1000,
        discardOldest: true,
        queueSize: 10
      });
    monitoredItem.on("changed", emitData(nodeId));
  });
};

I hope this makes a little more sense 😅 Thanks!

biancode commented 5 years ago

@erossignon what do you say, please?

biancode commented 5 years ago

on recreating the session I got sometimes two sessions - so node-opcua is here to handle that - I think it is the best to upgrade to v0.5+ for this

anthrogan commented 5 years ago

From what I understand there is also a session re-connection timeout also, so if there is a long gap between re-connection (Lets just say like a big Win Scr2012R2 Restart like my issue) then we are having to recreate the session as its timed out trying to recreate itself, a quick reset would not demonstrate this fault! Hopefully this will be a thing of the past soon node-opcua/node-opcua#572

I can implement a workaround for now by using ping nodes, but thank you for taking the time to answer @biancode :smile:

biancode commented 5 years ago

@anthrogan please, could you record some more logs for me with v3.8.0 and these settings to DEBUG?

DEBUG=client_base,opcua_client,client_session,client_subscription,reconnection,opcuaIIoT:connect* node red.js --set_max_old_space_size=512

anthrogan commented 5 years ago

will do bud, I'll get back to you with logs shortly.

Edit... nevermind, running it now, there's a slight difference in that this is running on windows and my production app is a container on linux but should be fine!

psorowka commented 5 years ago

@anthrogan @biancode I see no progress on node-opcua/node-opcua#572 so far but here is the workaround I am currently using:

  /* This workaround injects an additional event to the opcua client which is
   * emitted after sessions are activated or repaired after a connection break.
   * This workaround should not be required anymore as soon as a fix for
   * https://github.com/node-opcua/node-opcua/issues/572 has been published
   */
  OPCUAClient.prototype._on_connection_reestablished = function (callback) {
    const repairClientSessions = require('node-opcua-client/src/reconnection')
      .repair_client_sessions
    repairClientSessions(this, err => {
      if (!err) {
        this.emit('sessions_reestablished')
      }
      callback(err)
    })
  }

biancode commented 5 years ago

we start to test with v0.5.8 of node-opcua and to fix this

biancode commented 5 years ago

works better with 0.6.0 of node-opcua see v3.9+

psorowka commented 5 years ago

@biancode I see progress on node-opcua in 0.6.0 and 0.7.0 but unfortunately not in the session handling. all updates I have seen refer to the connection handling, but as far as I see, this it not the problem here..

anthrogan commented 5 years ago

reconnections are improved certainly but from time to time there is still the odd reconnection issue.

It is difficult to reproduce as this requires realistically catastrophic failure of a server and for that server to have considerable downtime! As @psorowka mentions there is progress on connection handling in node-opcua e.g. node-opcua/node-opcua#606 but still if session not closed correctly we will get an attempt to recreate the session on top of the existing one.

nodeopc is a behemoth of a repo, I suspect this fix isn't as straightforward as it seems!

biancode commented 5 years ago

@anthrogan as @erossignon told me, node-opcua wants to handle the whole reconnecting as the UAExpert do if I remember correctly.

araisch commented 5 years ago

as @erossignon told me, node-opcua wants to handle the whole reconnecting as the UAExpert do if I remember correctly.

That's a nice claim, but isn't working for me either. In my opinion this is a very complex thing, you encounter different behavior with different OPC-UA-Devices. Especially if you have to deal with "real hardware". Did some testing with different versions already and the best solution is still: If device replies ping, then try OPC-UA-Read, if there is no answer from read: delay and restart Node-Red until it works. Then start your productive flow. So you always get a clean status on startup.

erossignon commented 5 years ago

@araisch reconnection mechanism is fully specified and described in the OPCUA Specification and should really be implemented at the stack level. However, the application that uses the stack shall be aware when the connection is lost and shall restrict itself from trying to communicate with the device until node-opcua has managed to reestablish the connection and repair the existing subscription+monitored items. Some tasks are on the node-opcua backlog to introduce a extra event when the session is reestablished https://github.com/node-opcua/node-opcua/issues/572 , but not planned yet.

araisch commented 5 years ago

@erossignon Sounds very helpful, looking forward to that. Thanks for your great work at node-opcua :) It will be much easier if there is a full feedback about connection status from node-opcua instead of coding a state machine by myself, dealing with all possibilities. But it has to be reliable and transparent. A flow in a stand-alone-device somewhere abroad I can't just "deploy full" manually when it's hanging up.

mma-mm commented 5 years ago

I'm also experiencing this issue. Is there a workaround?

biancode commented 5 years ago

@mma-mm https://github.com/BiancoRoyal/node-red-contrib-iiot-opcua/issues/81#issuecomment-455915772 and it will be fixed with the next Node-RED v1.x contrib package

erossignon commented 5 years ago

@psorowka , I have a suggestion . why not propose a PR with your patch associated with a unit test so it can be incorporated into node-opcua ? Would you volunteer ?

biancode commented 4 years ago

I tested again with prerelease 3.11.1-4 and it worked very well based on node-opcua 0.7.4. reopen if you have trouble again.

laura-mc commented 4 years ago

I am having this issue again. Do you know how to solve the problem of the establishing the connection again? In my case, I am connecting to an external OPC UA server that whenevers it receives new data it is restarted, so, my node does not connect again to the server and node-red emits and error, and I need to reboot the device.

biancode commented 4 years ago

@laura-mc please provide some logs about that! Set or export DEBUG=opcuaIIoT* before you call node-red. A log file would be great, please.

laura-mc commented 4 years ago

@laura-mc please provide some logs about that! Set or export DEBUG=opcuaIIoT* before you call node-red. A log file would be great, please.

Hi @biancode, thanks for your quick reply. In fact, I am not using node-red-contrib-iiot-opcua, but node-red-contrib-opcua, but I think the problem comes from node-opcua.

Attached the errors. Error BadSessionInvalid.pdf

biancode commented 4 years ago

@laura-mc then, please use node-red-contrib-iiot-opcua

biancode commented 4 years ago

@laura-mc, you could test more with adds of file names from node-opcua like DEBUG=client_session,client_base and so on

BiancoRoyal / node-red-contrib-iiot-opcua