Mothership freezes with some large problem sizes, deadlocking the system

[heliosfa]

Background

On 1.0.0-alpha, the Mothership has a propensity for freezing with some larger problems on the 8-box system.

This manifests with the Reactive application when there are 50 nodes and 1001 timesteps (50050 devices) and the application is constrained to two devices per thread, giving 25025 threads total. The application seems to "lock up" at random points during execution and the Orchestrator hangs on exit. No instrumentation is produced.

Everything 50 nodes and larger fails with the same symptoms. Constraining the 50-node problem to three devices per thread works consistently.

Interestingly, 40 nodes (40040 devices) constrained to one device-per-thread (40040 threads) works while 80 nodes (80080) constrained to two devices-per-thread (40040 threads) does not.

Digging

Initially, I was under the impression that this was an issue within the Softswitch and spent a lot of time trying to debug it. As time went on, it was looking more and more like a Mothership issue to @mvousden took on the torch.

@mvousden found that the Backend input broker thread is hanging waiting for a mutex lock (the Debug input broker thread was also NOT hanging when it should have done). The mutex was not being release by the Backend output broker thread as it was stuck in a blocking send.

Further digging revealed that the Mothership was in the process of sending the barrier release packets when it became blocked. This stopped the Mothership from servicing received packets, causing the network to backup. This meant that the Backend output broker never sent the packet that it was trying to.

In other words, a perfect storm for a network deadlock.

TL;DR: Mothership became blocked sending barrier release packets, application floods network with actual traffic (inc. the Mothership), everything gets stuck. We did not go to the moon today.

The way forward

@mvousden has identified a couple of ways forward:

1. Increase the Softswitch barrier release delay (the time the softswitch waits after receiving a barrier release packet before moving into the main loop) to 50,000 loop iterations. This number was plucked out of the air.
2. Change the Send in the Backend output broker to a TrySend and sugar coat it in some back-off logic.

Both of these get the problem application running. The first completes in ~1s, the second in ~0.7s when used independently.

I have some further suggestions to improve this state of things:

• Modify Softswitch to use the hardware synchronisation barrier after barrier release. This has the benefit of scaling the delay based on the actual problem rather than being a guess, but ti will not work in multi-application runs.
• Multicase the barrier release packets. Currently, the barrier release packets are sent as unicast. This is A LOT of packets to send. If we could multicast these to each mailbox, we would reduce the number of packets by a factor of 64 (from 49152 to 768 in a full system run). Unfortunately, Hostlink does not currently support Local Multicast.
• Multi-mothership. This parallelises the barrier release and reduces the number of barrier packets each Mothership has to send. This introduces its of synchronisation issues and is essentially just kicking the can down the road.

More to come.

[mvousden]

Note that #248 addresses the second point in @heliosfa 's explanation, and not the first. From testing, we found that the second suggestion alone was sufficient for the case that was causing the trouble, but we should be aware of this issue for the future (which is why it remains open).

[heliosfa]

Multicast the barrier release packets.

I have been digging into this bit a little more to see what we can do. As expected, Hostlink does not expose the local multicast functionality. Hostlink does expose programmable routers, but the documentation on using them is incomplete.

I had a chat with @jordmorr about how he is doing it for Imputation and he helpfully walked me through things. Jordan has done this in raw tinsel at https://github.com/POETSII/jordmorr-tinsel/blob/multtests/apps/imputation/run.cpp

Interesting lines:

• 43: Prog router mesh creation
• 87 & 87: two key members of a struct
• 93: Destination has a kind, need to spec correct record type
• 95: The first 16 bits that get overridden
• 114: set the destination mbox
• 120: set the thread mask for the 64 destination threads on each mbox
• 122: set dest record type
• 124: append destination into a c++ sequence
• 201: spits out the routing key
• 574: write the routing table back out

This is all done after you have loaded the binaries but before everything is started (with StartAll in Jordan's case).

The issue with using programmable routers is that the multicast record always replace the least-significant 16-bits of the packet received by the the threads with a value specified in the routing record. This will stop us having a single general multicast destination that is usable for other purposes, unless we tweak the packet format to ensure that the device index is in the lowest 16-bits and that we always set it to FF.