Packets with "Invalid application ID" trigger a seg fault

[heliosfa]

Preamble This occurs on the BUGFIX-0285-HardwareIdleBarrier branch with FEATURE-0242-HardwareIdle-Mothership merged in locally. It may occur on other branches.

I have only seen the packet issue that causes this on Ayres - there may be an underlying issue there, but it should not lead to us seg faulting..

Issue After a number of runs, the Orchestrator will occasionally report that a packet with an "Invalid application ID" has been received and seg faults. Sometimes it is a plain signal 11, other times it is an std::bad_alloc signal 6.

Example application that inconsistently triggers the issue calc.xml, call file calc.poets and a bypass call file calcby.poets (remove .txt from all three. The bypass call file will only work after the normal one is used for a successful run)

Example output with the seg fault:

gmb@ayres:~/Orchestrator$ ./orchestrate.sh -b calc.poets
POETS> 02:24:23.01:  20(I) The microlog for the command 'load /engine = "../Config/POETSHardwareOneBox.ocfg"' will be written to '../Output/Microlog/Microlog_2021_11_02T02_24_23p0.plog'.
POETS> 02:24:23.01: 140(I) Topology loaded from file ||../Config/POETSHardwareOneBox.ocfg||.
POETS> 02:24:23.01:  20(I) The microlog for the command 'call /file = "/home/gmb/Orchestrator/calc.poets"' will be written to '../Output/Microlog/Microlog_2021_11_02T02_24_23p1.plog'.
POETS> 02:24:23.01:  20(I) The microlog for the command 'exit /at = stop' will be written to '../Output/Microlog/Microlog_2021_11_02T02_24_23p2.plog'.
POETS> 02:24:23.01:  69(I) Exit trigger staged - will exit when the next application is completely stopped.
POETS> 02:24:25.30:  20(I) The microlog for the command 'load /app = +calc.xml' will be written to '../Output/Microlog/Microlog_2021_11_02T02_24_23p3.plog'.
POETS> 02:24:25.30: 235(I) Application file ../calc.xml loading...
POETS> 02:24:25.30:  65(I) Application file ../calc.xml loaded in 2306 ms.
POETS> 02:24:25.37:  20(I) The microlog for the command 'tlink /app = *' will be written to '../Output/Microlog/Microlog_2021_11_02T02_24_25p0.plog'.
POETS> 02:24:25.37: 234(I) Typelinking graph instance 'calc'...
POETS> 02:24:25.37: 249(I) Successfully typelinked graph instance 'calc'.
POETS> 02:24:25.46:  20(I) The microlog for the command 'place /spread = *' will be written to '../Output/Microlog/Microlog_2021_11_02T02_24_25p1.plog'.
POETS> 02:24:25.46: 309(I) Attempting to place graph instance 'calc' using the 'spre' method...
POETS> 02:24:25.46: 302(I) Graph instance 'calc' placed successfully.
POETS> 02:28:28.98:  20(I) The microlog for the command 'compose /app = *' will be written to '../Output/Microlog/Microlog_2021_11_02T02_24_25p2.plog'.
POETS> 02:28:28.98: 803(I) Composing graph instance 'calc'...
POETS> 02:28:28.98: 804(I) Graph instance 'calc' composed successfully.
POETS> 02:28:28.01:  20(I) The microlog for the command 'deploy /app = *' will be written to '../Output/Microlog/Microlog_2021_11_02T02_28_28p0.plog'.
POETS> 02:28:28.01: 184(I) Deployment of graph instance 'calc' staged. Waiting for Mothership(s) to acknowledge receipt in the background.
POETS> 02:28:28.01:  20(I) The microlog for the command 'initialise /app = *' will be written to '../Output/Microlog/Microlog_2021_11_02T02_28_28p1.plog'.
POETS> 02:28:28.01: 187(I) Initialisation of graph instance 'calc' staged. Waiting for Mothership(s) to acknowledge receipt in the background.
POETS> 02:28:28.01:  20(I) The microlog for the command 'run /app = *' will be written to '../Output/Microlog/Microlog_2021_11_02T02_28_28p2.plog'.
POETS> 02:28:28.01: 188(I) Run of graph instance 'calc' staged. Waiting for Mothership(s) to acknowledge receipt in the background.
POETS> 02:28:28.23: 529(I) Mothership (rank 2): Deployment of application 'Arith::calc' (to this Mothership) complete.
POETS> 02:28:28.23: 530(I) Mothership (rank 2): Initialising fully-defined application 'Arith::calc'.
POETS> 02:28:28.23: 186(I) Application 'Arith::calc' successfully deployed on all Motherships it is mapped to.
POETS> 02:28:35.83: 531(I) Mothership (rank 2): Initialisation of application 'Arith::calc' (to this Mothership) complete.
POETS> 02:28:35.83: 532(I) Mothership (rank 2): Starting (running) fully-initialised application 'Arith::calc'.
POETS> 02:28:35.83: 186(I) Application 'Arith::calc' ready to start on all Motherships it is mapped to.
POETS> 02:28:35.83: 186(I) Application 'Arith::calc' running on all Motherships it is mapped to.
POETS>PROBLEM COMPLETE in 116 seconds (116.296s)
 02:30:31.04: 526(I) Mothership: Supervisor for application 'Arith::calc' has requested it to be stopped.
POETS> 02:30:31.04: 189(I) Stop of graph instance 'calc' staged. Waiting for Mothership(s) to acknowledge receipt in the background.
POETS> 02:30:31.05: 533(I) Mothership (rank 2): Stopping application 'Arith::calc' (which has been started).
POETS> 02:30:31.06: 186(I) Application 'Arith::calc' stopped on all Motherships it is mapped to.
POETS> 02:30:31.06:  68(I) Exit trigger hit - closing down the Orchestrator. The prompt is disabled.
POETS>gmb@ayres:~/Orchestrator$ ./orchestrate.sh -b calc.poets
POETS> 02:32:25.01:  20(I) The microlog for the command 'load /engine = "../Config/POETSHardwareOneBox.ocfg"' will be written to '../Output/Microlog/Microlog_2021_11_02T02_32_25p0.plog'.
POETS> 02:32:25.01: 140(I) Topology loaded from file ||../Config/POETSHardwareOneBox.ocfg||.
POETS> 02:32:25.01:  20(I) The microlog for the command 'call /file = "/home/gmb/Orchestrator/calc.poets"' will be written to '../Output/Microlog/Microlog_2021_11_02T02_32_25p1.plog'.
POETS> 02:32:25.01:  20(I) The microlog for the command 'exit /at = stop' will be written to '../Output/Microlog/Microlog_2021_11_02T02_32_25p2.plog'.
POETS> 02:32:25.01:  69(I) Exit trigger staged - will exit when the next application is completely stopped.
POETS> 02:32:27.30:  20(I) The microlog for the command 'load /app = +calc.xml' will be written to '../Output/Microlog/Microlog_2021_11_02T02_32_25p3.plog'.
POETS> 02:32:27.30: 235(I) Application file ../calc.xml loading...
POETS> 02:32:27.30:  65(I) Application file ../calc.xml loaded in 2307 ms.
POETS> 02:32:27.37:  20(I) The microlog for the command 'tlink /app = *' will be written to '../Output/Microlog/Microlog_2021_11_02T02_32_27p0.plog'.
POETS> 02:32:27.37: 234(I) Typelinking graph instance 'calc'...
POETS> 02:32:27.37: 249(I) Successfully typelinked graph instance 'calc'.
POETS> 02:32:27.46:  20(I) The microlog for the command 'place /spread = *' will be written to '../Output/Microlog/Microlog_2021_11_02T02_32_27p1.plog'.
POETS> 02:32:27.46: 309(I) Attempting to place graph instance 'calc' using the 'spre' method...
POETS> 02:32:27.46: 302(I) Graph instance 'calc' placed successfully.
POETS> 02:36:28.97:  20(I) The microlog for the command 'compose /app = *' will be written to '../Output/Microlog/Microlog_2021_11_02T02_32_27p2.plog'.
POETS> 02:36:28.97: 803(I) Composing graph instance 'calc'...
POETS> 02:36:28.97: 804(I) Graph instance 'calc' composed successfully.
POETS> 02:36:29.00:  20(I) The microlog for the command 'deploy /app = *' will be written to '../Output/Microlog/Microlog_2021_11_02T02_36_28p0.plog'.
POETS> 02:36:29.00: 184(I) Deployment of graph instance 'calc' staged. Waiting for Mothership(s) to acknowledge receipt in the background.
POETS> 02:36:29.00:  20(I) The microlog for the command 'initialise /app = *' will be written to '../Output/Microlog/Microlog_2021_11_02T02_36_29p0.plog'.
POETS> 02:36:29.00: 187(I) Initialisation of graph instance 'calc' staged. Waiting for Mothership(s) to acknowledge receipt in the background.
POETS> 02:36:29.00:  20(I) The microlog for the command 'run /app = *' will be written to '../Output/Microlog/Microlog_2021_11_02T02_36_29p1.plog'.
POETS> 02:36:29.00: 188(I) Run of graph instance 'calc' staged. Waiting for Mothership(s) to acknowledge receipt in the background.
POETS> 02:36:29.20: 529(I) Mothership (rank 2): Deployment of application 'Arith::calc' (to this Mothership) complete.
POETS> 02:36:29.20: 530(I) Mothership (rank 2): Initialising fully-defined application 'Arith::calc'.
POETS> 02:36:29.20: 186(I) Application 'Arith::calc' successfully deployed on all Motherships it is mapped to.
POETS> 02:36:35.86: 531(I) Mothership (rank 2): Initialisation of application 'Arith::calc' (to this Mothership) complete.
POETS> 02:36:35.86: 532(I) Mothership (rank 2): Starting (running) fully-initialised application 'Arith::calc'.
POETS> 02:36:35.86: 186(I) Application 'Arith::calc' ready to start on all Motherships it is mapped to.
POETS> 02:36:35.86: 186(I) Application 'Arith::calc' running on all Motherships it is mapped to.
POETS> 02:38:43.13: 520(W) Mothership: Invalid application ID '3f' in packet received from the compute fabric. Ignoring packet.
POETS>
===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 1985 RUNNING AT ayres
=   EXIT CODE: 139
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault (signal 11)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions
gmb@ayres:~/Orchestrator$

Example output of the std::bad_alloc:

gmb@ayres:~/Orchestrator$ ./orchestrate.sh -b calcby.poets
POETS> 05:01:29.01:  20(I) The microlog for the command 'load /engine = "../Config/POETSHardwareOneBox.ocfg"' will be written to '../Output/Microlog/Microlog_2021_11_02T05_01_29p0.plog'.
POETS> 05:01:29.01: 140(I) Topology loaded from file ||../Config/POETSHardwareOneBox.ocfg||.
POETS> 05:01:29.01:  20(I) The microlog for the command 'call /file = "/home/gmb/Orchestrator/calcby.poets"' will be written to '../Output/Microlog/Microlog_2021_11_02T05_01_29p1.plog'.
POETS> 05:01:29.01:  20(I) The microlog for the command 'exit /at = stop' will be written to '../Output/Microlog/Microlog_2021_11_02T05_01_29p2.plog'.
POETS> 05:01:29.01:  69(I) Exit trigger staged - will exit when the next application is completely stopped.
POETS> 05:01:31.63:  20(I) The microlog for the command 'load /app = +calc.xml' will be written to '../Output/Microlog/Microlog_2021_11_02T05_01_29p3.plog'.
POETS> 05:01:31.63: 235(I) Application file ../calc.xml loading...
POETS> 05:01:31.63:  65(I) Application file ../calc.xml loaded in 2638 ms.
POETS> 05:01:31.72:  20(I) The microlog for the command 'tlink /app = *' will be written to '../Output/Microlog/Microlog_2021_11_02T05_01_31p0.plog'.
POETS> 05:01:31.72: 234(I) Typelinking graph instance 'calc'...
POETS> 05:01:31.72: 249(I) Successfully typelinked graph instance 'calc'.
POETS> 05:01:32.82:  20(I) The microlog for the command 'place /spread = *' will be written to '../Output/Microlog/Microlog_2021_11_02T05_01_31p1.plog'.
POETS> 05:01:32.82: 309(I) Attempting to place graph instance 'calc' using the 'spre' method...
POETS> 05:01:32.82: 302(I) Graph instance 'calc' placed successfully.
POETS> 05:01:32.83:  20(I) The microlog for the command 'compose /bypass = *' will be written to '../Output/Microlog/Microlog_2021_11_02T05_01_32p0.plog'.
POETS> 05:01:32.83: 803(I) Bypassing graph instance 'calc'...
POETS> 05:01:32.83: 804(I) Graph instance 'calc' bypassed successfully.
POETS> 05:01:32.88:  20(I) The microlog for the command 'deploy /app = *' will be written to '../Output/Microlog/Microlog_2021_11_02T05_01_32p1.plog'.
POETS> 05:01:44.97: 184(I) Deployment of graph instance 'calc' staged. Waiting for Mothership(s) to acknowledge receipt in the background.
POETS> 05:01:44.97:  20(I) The microlog for the command 'initialise /app = *' will be written to '../Output/Microlog/Microlog_2021_11_02T05_01_32p2.plog'.
POETS> 05:01:44.97: 187(I) Initialisation of graph instance 'calc' staged. Waiting for Mothership(s) to acknowledge receipt in the background.
POETS> 05:01:44.97:  20(I) The microlog for the command 'run /app = *' will be written to '../Output/Microlog/Microlog_2021_11_02T05_01_32p3.plog'.
POETS> 05:01:44.97: 188(I) Run of graph instance 'calc' staged. Waiting for Mothership(s) to acknowledge receipt in the background.
POETS> 05:01:44.15: 529(I) Mothership (rank 2): Deployment of application 'Arith::calc' (to this Mothership) complete.
POETS> 05:01:44.15: 530(I) Mothership (rank 2): Initialising fully-defined application 'Arith::calc'.
POETS> 05:01:44.15: 186(I) Application 'Arith::calc' successfully deployed on all Motherships it is mapped to.
POETS> 05:01:52.63: 531(I) Mothership (rank 2): Initialisation of application 'Arith::calc' (to this Mothership) complete.
POETS> 05:01:52.63: 532(I) Mothership (rank 2): Starting (running) fully-initialised application 'Arith::calc'.
POETS> 05:01:52.63: 186(I) Application 'Arith::calc' ready to start on all Motherships it is mapped to.
POETS> 05:01:52.63: 186(I) Application 'Arith::calc' running on all Motherships it is mapped to.
POETS> 05:02:33.05: 518(W) Mothership: Received a packet for an external device - external support is not yet implemented. The software address of this packet is '80000003'. Ignoring packet.
POETS> 05:02:53.67: 518(W) Mothership: Received a packet for an external device - external support is not yet implemented. The software address of this packet is 'bb77a6ff'. Ignoring packet.
POETS> 05:02:53.09: 520(W) Mothership: Invalid application ID '39' in packet received from the compute fabric. Ignoring packet.
POETS>terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 9836 RUNNING AT ayres
=   EXIT CODE: 134
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions
gmb@ayres:~/Orchestrator$

[heliosfa]

Not directly related to this issue, but an interesting observation to do with Ayres. The spurious traffic is not limited to receiving packets with bad application IDs or packets for external devices, but we are also apparently getting random BARRIER packets while running and also spurious Debuglink traffic. Maybe it needs a reboot?

gmb@ayres:~/Orchestrator$ ./orchestrate.sh -b calc.poets
POETS> 05:13:49.01:  20(I) The microlog for the command 'load /engine = "../Config/POETSHardwareOneBox.ocfg"' will be written to '../Output/Microlog/Microlog_2021_11_02T05_13_49p0.plog'.
POETS> 05:13:49.01: 140(I) Topology loaded from file ||../Config/POETSHardwareOneBox.ocfg||.
POETS> 05:13:49.01:  20(I) The microlog for the command 'call /file = "/home/gmb/Orchestrator/calc.poets"' will be written to '../Output/Microlog/Microlog_2021_11_02T05_13_49p1.plog'.
POETS> 05:13:49.01:  20(I) The microlog for the command 'exit /at = stop' will be written to '../Output/Microlog/Microlog_2021_11_02T05_13_49p2.plog'.
POETS> 05:13:49.01:  69(I) Exit trigger staged - will exit when the next application is completely stopped.
POETS> 05:13:54.88:  20(I) The microlog for the command 'load /app = +calc.xml' will be written to '../Output/Microlog/Microlog_2021_11_02T05_13_49p3.plog'.
POETS> 05:13:54.88: 235(I) Application file ../calc.xml loading...
POETS> 05:13:54.88:  65(I) Application file ../calc.xml loaded in 4881 ms.
POETS> 05:13:54.05:  20(I) The microlog for the command 'tlink /app = *' will be written to '../Output/Microlog/Microlog_2021_11_02T05_13_54p0.plog'.
POETS> 05:13:54.05: 234(I) Typelinking graph instance 'calc'...
POETS> 05:13:54.05: 249(I) Successfully typelinked graph instance 'calc'.
POETS> 05:13:54.26:  20(I) The microlog for the command 'place /spread = *' will be written to '../Output/Microlog/Microlog_2021_11_02T05_13_54p1.plog'.
POETS> 05:13:54.26: 309(I) Attempting to place graph instance 'calc' using the 'spre' method...
POETS> 05:13:54.26: 302(I) Graph instance 'calc' placed successfully.
POETS> 05:18:04.34:  20(I) The microlog for the command 'compose /app = *' will be written to '../Output/Microlog/Microlog_2021_11_02T05_13_54p2.plog'.
POETS> 05:18:04.34: 803(I) Composing graph instance 'calc'...
POETS> 05:18:04.34: 804(I) Graph instance 'calc' composed successfully.
POETS> 05:18:04.38:  20(I) The microlog for the command 'deploy /app = *' will be written to '../Output/Microlog/Microlog_2021_11_02T05_18_04p0.plog'.
POETS> 05:18:04.38: 184(I) Deployment of graph instance 'calc' staged. Waiting for Mothership(s) to acknowledge receipt in the background.
POETS> 05:18:04.38:  20(I) The microlog for the command 'initialise /app = *' will be written to '../Output/Microlog/Microlog_2021_11_02T05_18_04p1.plog'.
POETS> 05:18:04.38: 187(I) Initialisation of graph instance 'calc' staged. Waiting for Mothership(s) to acknowledge receipt in the background.
POETS> 05:18:04.38:  20(I) The microlog for the command 'run /app = *' will be written to '../Output/Microlog/Microlog_2021_11_02T05_18_04p2.plog'.
POETS> 05:18:04.38: 188(I) Run of graph instance 'calc' staged. Waiting for Mothership(s) to acknowledge receipt in the background.
POETS> 05:18:05.63: 529(I) Mothership (rank 2): Deployment of application 'Arith::calc' (to this Mothership) complete.
POETS> 05:18:05.63: 530(I) Mothership (rank 2): Initialising fully-defined application 'Arith::calc'.
POETS> 05:18:05.63: 186(I) Application 'Arith::calc' successfully deployed on all Motherships it is mapped to.
POETS> 05:18:17.17: 531(I) Mothership (rank 2): Initialisation of application 'Arith::calc' (to this Mothership) complete.
POETS> 05:18:17.17: 186(I) Application 'Arith::calc' ready to start on all Motherships it is mapped to.
POETS> 05:18:17.17: 532(I) Mothership (rank 2): Starting (running) fully-initialised application 'Arith::calc'.
POETS> 05:18:17.17: 186(I) Application 'Arith::calc' running on all Motherships it is mapped to.
POETS> 05:21:25.25: 513(W) Mothership: Received an unexpected 'BARRIER' packet for an application in state 'RUNNING'. Ignoring.
POETS> 05:21:27.85: 513(W) Mothership: Received an unexpected 'BARRIER' packet for an application in state 'RUNNING'. Ignoring.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content '1'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'f1'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content '2e'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'fb'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content '1'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'fb'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content '44'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content '44'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'df'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'df'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'df'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'fd'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'eb'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'd4'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x20ed' with content 'c4'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x20ed' with content '30'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'fe'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ed'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content '17'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content '17'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content '17'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content '17'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'f7'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content '81'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ed'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content '17'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content '17'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content '17'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content '17'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'f7'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'f7'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'f7'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'fe'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'fe'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'fe'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'fe'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'fe'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'fe'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'f3'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'df'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'df'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'de'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'de'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'de'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'de'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'fe'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'fe'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'fd'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'fd'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'fd'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'fd'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'df'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'bf'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'bf'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'da'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'da'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'f4'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'f4'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'f4'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'f4'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content '7f'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content '7f'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'df'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content '7f'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'df'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'f6'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content '6d'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content '1f'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'f7'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'f7'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content '97'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'f5'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'f6'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'fd'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'fd'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'fd'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'fd'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'fd'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'fd'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'fd'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content '57'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content '57'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'bf'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'bf'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'ff'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content '34'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content '57'.
POETS> 05:21:41.41: 521(I) Debug byte received from thread '0x23ae' with content 'bf'.
POETS>

[mvousden]

A note for later - maybe there's some hysteresis here. Maybe the tinsels are still working after the Orchestrator finishes?

[heliosfa]

A note for later - maybe there's some hysteresis here. Maybe the tinsels are still working after the Orchestrator finishes?

That is what I was thinking at first, but the range of spurious output does not make sense for that.

• Barrier packets are only produced during the barrier, so why would there be some extras generated during a runaway application?
• We also do not make use of the debuglink for the spurious bytes there.
• Also, it is was a runaway application, none of the cores would have been running the bootloader and we would not have been able to load the new application.

[heliosfa]

Just to update with the details from today's E-Mails with @mn416, Ayres has a fan failure on one of the boards leading to overheating and is out of commission for the time being.

That said, we should still not be seg faulting in response.