Open hawkw opened 3 months ago
not just dump -l
, looks like any attempt to use the dump-agent on this system times out, since it's stuck talking to net.
eliza@niles ~ $ pfexec humility -t gimlet-c jefe -f hiffy
humility: attached to 0483:3754:000D00344741500820383733 via ST-Link V3
humility: successfully changed disposition for hiffy
eliza@niles ~ $ pfexec humility -t gimlet-c jefe -r hiffy
humility: attached to 0483:3754:000D00344741500820383733 via ST-Link V3
humility: successfully changed disposition for hiffy
eliza@niles ~ $ pfexec humility -t gimlet-c dump --all
humility dump failed: Probe could not be created
Caused by:
Probe was not found.
eliza@niles ~ $ pfexec humility -t gimlet-c dump --all
humility: attached to 0483:3754:000D00344741500820383733 via ST-Link V3
humility: using hiffy dump agent
humility dump failed: operation timed out
eliza@niles ~ $
Here's the core, for posterity.
The power sequencer is hung attempting to program the FPGA, which isn't responding by pulling CDONE low. This is blocking the net task from making progress, because it can't interact with its devices until power sequencing is complete. The dump-agent in turn is patiently waiting for that, for better or worse.
Here's the sequencer log:
humility: ring buffer drv_gimlet_seq_server::__RINGBUF in gimlet_seq:
TOTAL VARIANT
992167 Programming
1 Ice40PowerGoodV3P3(true)
1 IdentValid(false)
1 ChecksumValid(false)
1 Reprogram(true)
1 Ice40Rails
1 Ice40PowerGoodV1P2(true)
NDX LINE GEN COUNT PAYLOAD
0 264 1 1 Ice40Rails(true, true)
1 293 1 1 Ice40PowerGoodV1P2(true)
2 314 1 1 Ice40PowerGoodV3P3(true)
3 356 1 1 IdentValid(false)
4 359 1 1 ChecksumValid(false)
5 362 1 1 Reprogram(true)
6 376 1 65535 Programming
7 376 1 65535 Programming
8 376 1 65535 Programming
9 376 1 65535 Programming
10 376 1 65535 Programming
11 376 1 65535 Programming
12 376 1 65535 Programming
13 376 1 65535 Programming
14 376 1 65535 Programming
15 376 1 65535 Programming
16 376 1 65535 Programming
17 376 1 65535 Programming
18 376 1 65535 Programming
19 376 1 65535 Programming
20 376 1 65535 Programming
21 376 1 9142 Programming
And based on its interactions with the SPI driver, it's locking and unlocking without sending data:
fn Spi::lock() .................................................. 992167 calls
clients:
task gimlet_seq (0 restarts) ....................... = 0 ...... = 992167 ok
fn Spi::release() ............................................... 992166 calls
clients:
task gimlet_seq (0 restarts) ....................... = 0 ...... = 992166 ok
fn Spi::exchange() ................................................... 2 calls
clients:
task gimlet_seq (0 restarts) ....................... = 0 ........... = 2 ok
...which means, on review of the code, that it's failing this check:
// At this point, the iCE40 is _supposed_ to be chilling in programming mode
// listening for a bitstream. If this is the case it will be asserting
// (holding low) CDONE. Let's check!
if sys.gpio_read(config.cdone) != 0 {
// Welp, that sure didn't work.
return Err(Ice40Error::ChipNotListening);
}
...so it turns out somebody flashed a Gimlet SP image onto a PSC SP board, and that's what we're observing here. The PSC doesn't have a sequencer FPGA, so this is getting stuck waiting for an acknowledge from a missing chip. That part is working roughly as expected.
We don't currently have a consistent "board ID pins" scheme between boards that could catch this problem.
On reflection, this machine will have been asserting the SP boot failure ignition fault pin, which is working basically as designed. FWIW.
On a system with a whole lot of dumps (e.g. the gimlet from #1779, which has a
thermal
task that's crashed 7483 times), runninghumility dump -l
can time out. When this occurs, I've seen thehiffy
task left in a bad state where no other commands that usehiffy
can succeed.For instance:
manually restarting the
hiffy
task usinghumility jefe -fault hiffy
andhumility jefe --release hiffy
, seems to unstick it, until the next time that you runhumility dump -l
.