Luos-io / luos_engine

Open-source and real-time orchestrator for cyber-physical-systems, to easily design, test and deploy embedded applications and digital twins.
https://www.luos.io
MIT License
517 stars 55 forks source link

[BUG] PyLuos bootloader fails on the last node of a network having exactly 25 nodes + Gate #308

Closed fleborne closed 2 years ago

fleborne commented 2 years ago

Which version the bug has been detected Luos engine 2.5.2

Describe the bug When exactly 25 nodes (in addition to the one hosting the gate and the pipe) are on the network, pyluos-bootloader fails to flash the last one of them (node ID 26), stopping at 3% of progress, when the nodes have just been flashed with the boot_service through a ST-LINK probe. This behaviour is always the same, no matter if I try to flash all nodes together or only the last node. I can reboot the network, switch the boards in the network, ..., it is always node with ID 26 that will be stuck at 3% of flashing.

If I isolate a subsection of the network, let's say nodes with ID 20 to 26, by disconnecting the rest of the boards, I can then flash all 7 boards normally. When I put this subsection back into the network, everything then works perfectly: I can reprogram the whole network using pyluos-bootloader.

Network configuration The image below shows the network made of one Gate + Pipe (NUCLEO-G431KB with custom shield) and 25 custom nodes (STM32G431KB based), each one hosting one custom ServoMotor service.

The Gate + Pipe are powered through the USB cable. Each motor control node is powered (12 V regulated down to 3.3 V by each PCB) through the harness visible in the middle of the image (U shape).

IMG_20220802_154039

The list of nodes and services is the following:

  ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
  ┃  ╭node 1            /!\ Not certified            ┃
  ┃  │  Type                Alias               ID   ┃
  ┃  ├> Gate                gate                1    ┃
  ┃  ╰> Pipe                Pipe                2    ┃
╔>┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
║     ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
╚══ 1>┃0 ╭node 2            /!\ Not certified            ┃
      ┃  │  Type                Alias               ID   ┃
      ┃  ╰> ServoMotor          nb55                3    ┃
    ╔>┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
    ║     ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ╚══ 1>┃0 ╭node 3            /!\ Not certified            ┃
          ┃  │  Type                Alias               ID   ┃
          ┃  ╰> ServoMotor          nb551               4    ┃
        ╔>┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
        ║     ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
        ╚══ 1>┃0 ╭node 4            /!\ Not certified            ┃
              ┃  │  Type                Alias               ID   ┃
              ┃  ╰> ServoMotor          nb552               5    ┃
            ╔>┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
            ║     ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
            ╚══ 1>┃0 ╭node 5            /!\ Not certified            ┃
                  ┃  │  Type                Alias               ID   ┃
                  ┃  ╰> ServoMotor          nb553               6    ┃
                ╔>┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
                ║     ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
                ╚══ 1>┃0 ╭node 6            /!\ Not certified            ┃
                      ┃  │  Type                Alias               ID   ┃
                      ┃  ╰> ServoMotor          nb554               7    ┃
                    ╔>┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
                    ║     ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
                    ╚══ 1>┃0 ╭node 7            /!\ Not certified            ┃
                          ┃  │  Type                Alias               ID   ┃
                          ┃  ╰> ServoMotor          nb555               8    ┃
                        ╔>┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
                        ║     ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
                        ╚══ 1>┃0 ╭node 8            /!\ Not certified            ┃
                              ┃  │  Type                Alias               ID   ┃
                              ┃  ╰> ServoMotor          nb556               9    ┃
                            ╔>┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
                            ║     ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
                            ╚══ 1>┃0 ╭node 9            /!\ Not certified            ┃
                                  ┃  │  Type                Alias               ID   ┃
                                  ┃  ╰> ServoMotor          nb557               10   ┃
                                ╔>┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
                                ║     ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
                                ╚══ 1>┃0 ╭node 10            /!\ Not certified            ┃
                                      ┃  │  Type                Alias               ID   ┃
                                      ┃  ╰> ServoMotor          nb558               11   ┃
                                    ╔>┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
                                    ║     ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
                                    ╚══ 1>┃0 ╭node 11            /!\ Not certified            ┃
                                          ┃  │  Type                Alias               ID   ┃
                                          ┃  ╰> ServoMotor          nb559               12   ┃
                                        ╔>┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
                                        ║     ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
                                        ╚══ 1>┃0 ╭node 12            /!\ Not certified            ┃
                                              ┃  │  Type                Alias               ID   ┃
                                              ┃  ╰> ServoMotor          nb5510              13   ┃
                                            ╔>┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
                                            ║     ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
                                            ╚══ 1>┃0 ╭node 13            /!\ Not certified            ┃
                                                  ┃  │  Type                Alias               ID   ┃
                                                  ┃  ╰> ServoMotor          nb5511              14   ┃
                                                ╔>┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
                                                ║     ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
                                                ╚══ 1>┃0 ╭node 14            /!\ Not certified            ┃
                                                      ┃  │  Type                Alias               ID   ┃
                                                      ┃  ╰> ServoMotor          nb5512              15   ┃
                                                    ╔>┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
                                                    ║     ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
                                                    ╚══ 1>┃0 ╭node 15            /!\ Not certified            ┃
                                                          ┃  │  Type                Alias               ID   ┃
                                                          ┃  ╰> ServoMotor          nb5513              16   ┃
                                                        ╔>┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
                                                        ║     ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
                                                        ╚══ 1>┃0 ╭node 16            /!\ Not certified            ┃
                                                              ┃  │  Type                Alias               ID   ┃
                                                              ┃  ╰> ServoMotor          nb5514              17   ┃
                                                            ╔>┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
                                                            ║     ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
                                                            ╚══ 1>┃0 ╭node 17            /!\ Not certified            ┃
                                                                  ┃  │  Type                Alias               ID   ┃
                                                                  ┃  ╰> ServoMotor          nb5515              18   ┃
                                                                ╔>┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
                                                                ║     ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
                                                                ╚══ 1>┃0 ╭node 18            /!\ Not certified            ┃
                                                                      ┃  │  Type                Alias               ID   ┃
                                                                      ┃  ╰> ServoMotor          nb5516              19   ┃
                                                                    ╔>┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
                                                                    ║     ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
                                                                    ╚══ 1>┃0 ╭node 19            /!\ Not certified            ┃
                                                                          ┃  │  Type                Alias               ID   ┃
                                                                          ┃  ╰> ServoMotor          nb5517              20   ┃
                                                                        ╔>┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
                                                                        ║     ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
                                                                        ╚══ 1>┃0 ╭node 20            /!\ Not certified            ┃
                                                                              ┃  │  Type                Alias               ID   ┃
                                                                              ┃  ╰> ServoMotor          nb5518              21   ┃
                                                                            ╔>┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
                                                                            ║     ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
                                                                            ╚══ 1>┃0 ╭node 21            /!\ Not certified            ┃
                                                                                  ┃  │  Type                Alias               ID   ┃
                                                                                  ┃  ╰> ServoMotor          nb5519              22   ┃
                                                                                ╔>┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
                                                                                ║     ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
                                                                                ╚══ 1>┃0 ╭node 22            /!\ Not certified            ┃
                                                                                      ┃  │  Type                Alias               ID   ┃
                                                                                      ┃  ╰> ServoMotor          nb5520              23   ┃
                                                                                    ╔>┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
                                                                                    ║     ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
                                                                                    ╚══ 1>┃0 ╭node 23            /!\ Not certified            ┃
                                                                                          ┃  │  Type                Alias               ID   ┃
                                                                                          ┃  ╰> ServoMotor          nb5521              24   ┃
                                                                                        ╔>┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
                                                                                        ║     ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
                                                                                        ╚══ 1>┃0 ╭node 24            /!\ Not certified            ┃
                                                                                              ┃  │  Type                Alias               ID   ┃
                                                                                              ┃  ╰> ServoMotor          nb5522              25   ┃
                                                                                            ╔>┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
                                                                                            ║     ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
                                                                                            ╚══ 1>┃0 ╭node 25            /!\ Not certified            ┃
                                                                                                  ┃  │  Type                Alias               ID   ┃
                                                                                                  ┃  ╰> ServoMotor          nb5523              26   ┃
                                                                                                ╔>┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
                                                                                                ║     ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
                                                                                                ╚══ 1>┃0 ╭node 26            /!\ Not certified            ┃
                                                                                                      ┃  │  Type                Alias               ID   ┃
                                                                                                      ┃  ╰> ServoMotor          nb5524              27   ┃
                                                                                                     >┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

The Gate and Pipe are flashed with the code found in luos_engine's examples folder, only modified to match the hardware definition and the size of the network. Their _nodeconfig.h file is

#ifndef _NODE_CONFIG_H_
#define _NODE_CONFIG_H_

#define MAX_SERVICE_NUMBER 2
#define MSG_BUFFER_SIZE    8192
#define MAX_MSG_NB         80

#define TX_EN_PIN GPIO_PIN_15
#define TX_EN_PORT GPIOA

#define MAX_RTB_ENTRY            80
#define GATE_BUFF_SIZE           2048
#define PIPE_TO_LUOS_BUFFER_SIZE 2048
#define LUOS_TO_PIPE_BUFFER_SIZE 4096

#define INIT_TIME 150

#endif /* _NODE_CONFIG_H_ */

How to reproduce the bug

  1. using a ST-LINK v2 probe, flash the bootloader (from luos_engine examples) on each of the 25 boards
  2. flash all boards using pyluos-bootloader: pyluos-bootloader flash <port> -t {2..26} -b <firmware.bin>
  3. the process gets stuck at 3% of flashing the last node (ID 26)

Additional context

Salem-Tho commented 2 years ago

Fix in release 2.5.3