dmroeder / pylogix

Read/Write data from Allen Bradley Compact/Control Logix PLC's
Apache License 2.0
587 stars 181 forks source link

Forward Open Failed #249

Closed Corazon1110 closed 2 months ago

Corazon1110 commented 3 months ago

Type of issue

Description of issue

_ Hi. I got trouble like this: I use 1 device to connect to PLC Rockwell. PLC model

1769-L33ER/A LOGIX5333ER

So far, the connection with the PLC has been successful. However, recently, I've been using three distinct scripts on the same devices to simultaneously connect with the same PLC. While it's still able to establish a connection with the PLC and retrieve the PLC model, it's unable to fetch the PLC tags.

class RockwellPLC(PLC):
    def __init__(self):
        super().__init__()
        self.plc = PLC()
        PATH = os.path.join(Path(os.path.dirname(__file__)).parent, 'core')
        # Define the path to the JSON file
        json_file_path = os.path.join(PATH, "plc_config.json")
        with open(json_file_path, 'r') as json_file:
                config_data = json.load(json_file)
        plc_parameters = config_data.get("plc_parameters", {})
        self.IPAddress = plc_parameters.get("PLC_IP")
        self.ProcessorSlot = int(plc_parameters.get("PLC_SLOT"))

    def read_signal_station(self, station):

        self.Enable_array = [self.Read(f'PpST{station}.PpPrequisiteEnable[{i}]').Value for i in
                             range(15, 20)]

        self.Complete_array = [self.Read(f'PpST{station}.PpPrequisite[{i}]').Value for i in
                               range(15, 20)]

Expected behavior

2 array of Enable and Complete Signal


[False, False, False, False, False]
[False, False, False, False, False]

Actual behavior

sometimes it cannot connect to PLC and return:

[None, None, None, None, None]
[None, None, None, None, None]

issue type: forward open failed

Code

class PlcRockwell:
    def __init__(self):
        self.Station_Complete = None
        print('Initial PLC')
        self.ModelNumber = None
        self.Complete_array = list()
        self.Enable_array = list()
        self.plc = None
        self.VIN = None
        self.StationArray = None
        self._connect_plc()
        # self.append_object_list()

    def init_cam_string(self):
        self._NullInspString = "NULL|NULL|NULL|0"
        self._Current_Cam_1_Insp = self._NullInspString
        self._Current_Cam_2_Insp = self._NullInspString
        self._Current_Cam_3_Insp = self._NullInspString
        self._Current_Cam_4_Insp = self._NullInspString
        self._init_cam_file()
        self._Cam_1_list = list()
        self._Cam_2_list = list()
        self._Cam_3_list = list()
        self._Cam_4_list = list()

    def _connect_plc(self):
        PATH = os.path.dirname(__file__)
        # Define the path to the JSON file
        json_file_path = os.path.join(PATH, "param.json")
        try:
            # Load JSON content from the file
            with open(json_file_path, 'r') as json_file:
                config_data = json.load(json_file)

            # PLC Parameters
            plc_parameters = config_data.get("plc_parameters", {})
            self.PLC_IP, self.PLC_RACK, self.PLC_SLOT = (
                plc_parameters.get(key) for key in
                ["PLC_IP", "PLC_RACK", "PLC_SLOT"]
            )
            self.StationArray = plc_parameters.get("STATION")
            print(self.PLC_IP)
            try:
                ipaddress.IPv4Address(self.PLC_IP)
            except Exception as e:
                exit(f"Invalid PLC ID address format {e}")
        except Exception as e:
            exit(f'local_orch/_connect_plc: {e}')

    def get_data(self) -> None:
        """
        Retrieves data from the PLC.
        Reads VIN, ModelNumber, PP_Enable, and Insp_Compete from the PLC.
        """
        try:
            with PLC() as comm:
                comm.IPAddress = self.PLC_IP
                comm.ProcessorSlot = self.PLC_SLOT

                self.VIN = comm.Read(f'PpST{str(self.StationArray)}.FrPpModelSerial').Value
                self.ModelNumber = comm.Read(f'PpST{str(self.StationArray)}.FrPpModelNumber').Value
                self.Complete_array.clear()
                self.Enable_array.clear()
                self.Station_Complete = comm.Read(f'PpST{str(self.StationArray)}.PpStationComplete').Value
                for i in range(15, 20):
                    self.Enable_array.append(comm.Read(f'PpST{str(self.StationArray)}.PpPrequisiteEnable[{i}]').Value)
                    self.Complete_array.append(comm.Read(f'PpST{str(self.StationArray)}.PpPrequisite[{i}]').Value)
                if self.VIN == "":
                    self.VIN = "NULL"
                    self.ModelNumber = "NULL"

        except Exception as e:
            # Handle exceptions as needed
            print(f"Error: {e}")

Please provide a minimal, reproducible example.

https://stackoverflow.com/help/minimal-reproducible-example

Screenshots

Stacktrace

Versions

Version 1.0.1 Include versions to

kyle-github commented 3 months ago

It might be duplicate connection IDs. Is every simultaneous connection using a different connection ID? If the generation of connection IDs is not random enough, you can have more than one process generate the same ID.

For reasons unclear to me Rockwell decided that client connection IDs must be unique across all EIP sessions.

Best, Kyle

On Wed, Jul 3, 2024, 6:39 AM Corazon1110 @.***> wrote:

Type of issue

  • [ X] Bug

Description of issue

_ Hi. I got trouble like this: I use 1 device to connect to PLC Rockwell. PLC model

1769-L33ER/A LOGIX5333ER

So far, the connection with the PLC has been successful. However, recently, I've been using three distinct scripts on the same devices to simultaneously connect with the same PLC. While it's still able to establish a connection with the PLC and retrieve the PLC model, it's unable to fetch the PLC tags.

class RockwellPLC(PLC): def init(self): super().init() self.plc = PLC() PATH = os.path.join(Path(os.path.dirname(file)).parent, 'core')

Define the path to the JSON file

    json_file_path = os.path.join(PATH, "plc_config.json")
    with open(json_file_path, 'r') as json_file:
            config_data = json.load(json_file)
    plc_parameters = config_data.get("plc_parameters", {})
    self.IPAddress = plc_parameters.get("PLC_IP")
    self.ProcessorSlot = int(plc_parameters.get("PLC_SLOT"))

def read_signal_station(self, station):

    self.Enable_array = [self.Read(f'PpST{station}.PpPrequisiteEnable[{i}]').Value for i in
                         range(15, 20)]

    self.Complete_array = [self.Read(f'PpST{station}.PpPrequisite[{i}]').Value for i in
                           range(15, 20)]

Expected behavior

2 array of Enable and Complete Signal

[False, False, False, False, False] [False, False, False, False, False]

Actual behavior

sometimes it cannot connect to PLC and return:

[None, None, None, None, None] [None, None, None, None, None]

issue type: forward open failed Code

class PlcRockwell: def init(self): self.Station_Complete = None print('Initial PLC') self.ModelNumber = None self.Complete_array = list() self.Enable_array = list() self.plc = None self.VIN = None self.StationArray = None self._connect_plc()

self.append_object_list()

def init_cam_string(self):
    self._NullInspString = "NULL|NULL|NULL|0"
    self._Current_Cam_1_Insp = self._NullInspString
    self._Current_Cam_2_Insp = self._NullInspString
    self._Current_Cam_3_Insp = self._NullInspString
    self._Current_Cam_4_Insp = self._NullInspString
    self._init_cam_file()
    self._Cam_1_list = list()
    self._Cam_2_list = list()
    self._Cam_3_list = list()
    self._Cam_4_list = list()

def _connect_plc(self):
    PATH = os.path.dirname(__file__)
    # Define the path to the JSON file
    json_file_path = os.path.join(PATH, "param.json")
    try:
        # Load JSON content from the file
        with open(json_file_path, 'r') as json_file:
            config_data = json.load(json_file)

        # PLC Parameters
        plc_parameters = config_data.get("plc_parameters", {})
        self.PLC_IP, self.PLC_RACK, self.PLC_SLOT = (
            plc_parameters.get(key) for key in
            ["PLC_IP", "PLC_RACK", "PLC_SLOT"]
        )
        self.StationArray = plc_parameters.get("STATION")
        print(self.PLC_IP)
        try:
            ipaddress.IPv4Address(self.PLC_IP)
        except Exception as e:
            exit(f"Invalid PLC ID address format {e}")
    except Exception as e:
        exit(f'local_orch/_connect_plc: {e}')

def get_data(self) -> None:
    """
    Retrieves data from the PLC.
    Reads VIN, ModelNumber, PP_Enable, and Insp_Compete from the PLC.
    """
    try:
        with PLC() as comm:
            comm.IPAddress = self.PLC_IP
            comm.ProcessorSlot = self.PLC_SLOT

            self.VIN = comm.Read(f'PpST{str(self.StationArray)}.FrPpModelSerial').Value
            self.ModelNumber = comm.Read(f'PpST{str(self.StationArray)}.FrPpModelNumber').Value
            self.Complete_array.clear()
            self.Enable_array.clear()
            self.Station_Complete = comm.Read(f'PpST{str(self.StationArray)}.PpStationComplete').Value
            for i in range(15, 20):
                self.Enable_array.append(comm.Read(f'PpST{str(self.StationArray)}.PpPrequisiteEnable[{i}]').Value)
                self.Complete_array.append(comm.Read(f'PpST{str(self.StationArray)}.PpPrequisite[{i}]').Value)
            if self.VIN == "":
                self.VIN = "NULL"
                self.ModelNumber = "NULL"

    except Exception as e:
        # Handle exceptions as needed
        print(f"Error: {e}")

Please provide a minimal, reproducible example.

https://stackoverflow.com/help/minimal-reproducible-example Screenshots Stacktrace Versions

Version 1.0.1 Include versions to

  • pylogix: 1.0.1
  • plc model: 1769-L33ER/A LOGIX5333ER
  • python: 3.8,10
  • OS: Linux

— Reply to this email directly, view it on GitHub https://github.com/dmroeder/pylogix/issues/249, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAN4LC7NCDJ3D3VDQKFO2P3ZKN6BDAVCNFSM6AAAAABKIYMH2WVHI2DSMVQWIX3LMV43ASLTON2WKOZSGM4DONRVGM4DMOA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

Corazon1110 commented 3 months ago

simultaneous

Hi kyle, how to I get these IDs? And how to make them unique? thanks

kyle-github commented 3 months ago

This is more a question for @dmroeder but he's on vacation. I am not sure how to get pylogix to tell you what it is using. Maybe @thefern knows?

On Thu, Jul 4, 2024, 5:06 AM Corazon1110 @.***> wrote:

simultaneous

Hi kyle, how to I get these IDs? And how to make them unique? thanks

— Reply to this email directly, view it on GitHub https://github.com/dmroeder/pylogix/issues/249#issuecomment-2207956894, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAN4LC32QSUBX6EZ7W2QAETZKS33PAVCNFSM6AAAAABKIYMH2WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBXHE2TMOBZGQ . You are receiving this because you commented.Message ID: @.***>

TheFern commented 3 months ago

@TheFern2 see above

TheFern2 commented 3 months ago

As far as I can tell conn ids are generated randomly. I'm thinking if the random module uses time as seed and scripts start at the exact same time it is possible ids are duplicated. Can you confirm if scripts are launched at the same time? Or add a print statement to print the id on the function below.

https://github.com/dmroeder/pylogix/blob/master/pylogix/lgx_comm.py#L305C5-L305C26

TheFern2 commented 3 months ago

I would delay scripts starting by one ms if they're are starting simultaneously,that should change the seed which is time based on the random module.

kyle-github commented 3 months ago

Sorry about the wrong "Fern"!

If you can print out the Forward Open parameters and response we can see if it is the PLC returning an error and if so what it was. There is a specific error that is returned when there is a duplicate connection ID.

I am traveling so I don't have easy access to choose on GitHub.

Best, Kyle

On Fri, Jul 5, 2024, 1:17 AM Fernando B @.***> wrote:

I would delay scripts starting by one ms if they're are starting simultaneously,that should change the seed which is time based on the random module.

— Reply to this email directly, view it on GitHub https://github.com/dmroeder/pylogix/issues/249#issuecomment-2209635024, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAN4LC76TTSLXZZ36Y6LTNTZKXJZNAVCNFSM6AAAAABKIYMH2WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBZGYZTKMBSGQ . You are receiving this because you commented.Message ID: @.***>

TheFern2 commented 3 months ago

I'm 99% positive the id is duplicated if the scripts are launched at the same time as he did stated "simultaneously" running 3 scripts. Random uses time as the seed by default.

The only way to prevent same ids on scripts starting at the same time would be to add a seed argument for the PLC constructor. Unless there's a way to internally check if the id already exists and up it by one.

dmroeder commented 3 months ago

The connection ID is generated when the first read (among other thing) is made. You might be able to prove the connection ID idea by printing it immediately after the first read: print(self.comm.conn._ot_connection_id)

Is the issue consistent? What PLC/Ethernet module is involved? < Never mind, I see it is a L33ER.

Never mind, the connection ID is not exposed.

kyle-github commented 3 months ago

From what the OP said, it seems to fail intermittently at the Forward Open stage do the read would not get anywhere.

On Sat, Jul 6, 2024, 8:21 AM dmroeder @.***> wrote:

The connection ID is generated when the first read (among other thing) is made. You might be able to prove the connection ID idea by printing it immediately after the first read: print(self.comm.conn._ot_connection_id)

Is the issue consistent? What PLC/Ethernet module is involved?

— Reply to this email directly, view it on GitHub https://github.com/dmroeder/pylogix/issues/249#issuecomment-2211678098, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAN4LC2Z5Y2G2B5YDHSE333ZK6EH7AVCNFSM6AAAAABKIYMH2WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJRGY3TQMBZHA . You are receiving this because you commented.Message ID: @.***>

dmroeder commented 3 months ago

From what the OP said, it seems to fail intermittently at the Forward Open stage do the read would not get anywhere.

Read starts the connection process, so even though it fails, the connection ID should have been generated.

dmroeder commented 3 months ago

@Corazon1110, I see you are setting processor slot, even though this is a CompactLogix. Have you tried removing that? I'm wondering if the slot is being set to something other that 0 occasionally, this would cause a forward open fail.

Corazon1110 commented 2 months ago

I will try that and give feedback. Thank you all for your support.

Corazon1110 commented 2 months ago

After a period of monitoring, I think the problem is in the PLC. The PLC is communicating with so many devices that it seems to be overloaded. Currently I have 4 computers communicating with the PLC (Of course there are many other peripheral devices connected to the PLC). When the error occurs, all computers cannot read the tag on the PLC even though it still works. Read PLC properties. Do you think this is the right conclusion?

dmroeder commented 2 months ago

The best information for diagnosing this is the PLC's web page. You can look at packet statistics and utilization. Put the PLC's IP address in the web browser. Diagnostic Overview and Ethernet Statistics will tell you a lot.

Corazon1110 commented 2 months ago

HI, what information do I need to focus on? Rockwell

dmroeder commented 2 months ago

Ooh, MAC transmit errors out Out Errors are a problem. 0 is the only acceptable number for a properly configured network. It is possible they are unrelated to your problem, but they are a problem.

Can you post Diagnostic Overview and Application Connections?

Corazon1110 commented 2 months ago

Here are the information: image image

dmroeder commented 2 months ago

Only you know your system so don't make any changes if it is safe to do so and you are comfortable.

I see a number of problems. First off is the CPU utilization is very high. You have several devices configured for multi-cast, they should be unicast unless there is a very good reason otherwise. Missed packets should be zero. I would be surprised if you weren't already having random connection issues with devices in the tree, especially the ones with missed packets.

My experience with missed packets is that devices are not set to auto negotiate.

All those devices that are configured for 10ms, do the really need to be 10ms? If they don't need 10ms update rate, I'd extend them to reduce load CPU load.

To pick a couple of devices to look at, what is .136 and what is .123?

Corazon1110 commented 2 months ago

Thank you. Those are compressor solenoid. I will take a look and see if I can improve it. And of course, I have many trouble with connection recently, and now I know why 👍 thank you!

kyle-github commented 2 months ago

Ooph... As @dmroeder said, this is very high usage. Especially for a CompactLogix. As far as I understand, on a CompactLogix, the main CPU is doing all the work of running the PLC code and the network code.

UDP-based connections may use fewer resources than TCP-based ones (certainly for the buffer window) but you do get packet loss. Even on a fairly lightly loaded network, you will see some UDP packet loss.

The total number of connections looks pretty high too. According to one of the screens you show, you are using nearly 80% of all the CIP connection resources. I would aim for closer to 50%.

My rules of thumb for loads are:

  1. never load the network CPU (if it is a separate module) more than 75%. If it is a shared module, this should be closer to 30% for the network processing load.
  2. Keep the connection resource usage as low as you can go. One thing that can make connection resource usage really painful is if you have code that is constantly connecting, dropping the connection, and reconnecting again. From what I have seen, I think that the clean up of some internal connection resources may be asynchronous to the closing of the connection. I.e. you close the connection and some time later the PLC cleans up the internal resources. This means that you may actually be running out of connection resources even though it looks like you have 20% left.
  3. For high data rates, prefer IO or produced/consumed tags over explicit messaging. UDP-based data flow is a lot lighter weight than TCP-based flow. That said, you will get packet loss with UDP. So make sure that your system is designed with that in mind.
  4. If using multicast/broadcast, make sure that your ACK rate is lower than your data rate. This can be asymmetrical. I.e. you can have data coming in every 20ms, but only ACK every 100.
  5. Make sure every part of your network is using full duplex. As @dmroeder notes, it is bad if you have a mismatch.

Multicast can help bandwidth and latency considerably if you have multiple clients hung off of each router/switch. The routers will work together to make sure that a single stream of multicast (UDP) data goes to each router/switch and then internally is fed to all the local clients. But be aware that a multicast connection effectively consumes 2 connections as far as I can tell. The docs say that you can set up the connection with one TCP-based connection then drop the TCP part. From what I read in the field, this does not always work and so most leave the TCP connection up and use it for ACKs.

Corazon1110 commented 2 months ago

Many thanks to you guys <3. I changed the RPI from 10 ms to 60 ms, and the network CPU dropped to around 25-25%, that's a huge difference. There is no issue with your code and problem have been solved, so I will close this issue.