[Bug] SPIKE/Inventor Hub keeps turning off completely during execution. (power does not stay on during watchdog timer reset)

[TheVinhLuong102]

David, Laurens,

We are having trouble running the our Birthday Cake Cutter program reliably for a long time. The Inventor Hub keeps turning off after about a minute. It doesn't just exit the program, but turns off completely, and we have to boot it up again every time.

Is it a sign of hitting some memory limit on this hub? Is the concerned program considered big? What can we do to debug this further?

Thank you very much!

Vinh

[laurensvalk]

Could it be time to charge the battery? Does the light blink orange? Does it happen with other programs or just this one?

[TheVinhLuong102]

Laurens,

We checked and the battery was 100% charged (we did make sure by charging the battery to "green" again).

There is no warning sign / indicator that the hub would turn off soon -- it just goes.

This phenomenon is happening to just this one program. This is probably the longest one (~200 LOCs) we've written so far for the SPIKE/Inventor hub.

Vinh

[TheVinhLuong102]

The program size sent to the hub is 3.15kb, as measured by pybricksdev.

[TheVinhLuong102]

Let me reinstall the firmware -- both official and Pybricks -- and see if this problem remains.

[TheVinhLuong102]

I have reinstalled the firmware and can confirm that the problem still persists. :|

[laurensvalk]

Thanks for the additional information.

It's a good thing that it persists after reinstallation. That means it's consistent so it's easier to reproduce and fix.

It's possible that the hub got stuck at some point. We automatically turn the hub off when this happens. This is much more convenient than removing the battery.

To confirm, you can print the hub.system.reset_reason() after this happens. This is already documented in the latest beta - let me know if you need an example program.

It would also be great if you have a reproducible case that does not require building the robot. Is it possible to reproduce this with just the hub and motors/sensors attached to it?

[TheVinhLuong102]

Laurens,

Yes the reproducibility is indeed a good opportunity for debugging.

Could you point me to an example of invoking the hub.system.reset_reason()?

Vinh

[dlech]

The Prime/Inventor hubs are not able to reboot without losing power, so you will probably need to do this with USB plugged in to keep the power on.

Then just print('reason:', hub.system.reset_reason()) to get the reason.

[TheVinhLuong102]

David, when should we trigger that call? Is there a condition that we should catch?

[dlech]

Just run it as a standalone program after the hub reboots itself.

[TheVinhLuong102]

David, is the following debugging flow correct?

1. Run our own program (e.g. Birthday Cake Cutter) through Bluetooth using pybricksdev run ble
2. Operate the robot until the hub turns off
3. Plug the hub into the laptop with the USB cable
4. Manually turn on the hub, into the Pybricks mode
5. Run a program with print('reason:', hub.system.reset_reason()) though USB using pybricksdev run usb
6. Get the reset_reason() result as an integer number

[TheVinhLuong102]

And, by the way, could you point us to the beta doc site? The stable doc site https://docs.pybricks.com/en/stable/ doesn't mention these features, am I right?

[laurensvalk]

You can select a version of the documentation in the lower left corner. Stable is the default, but the Luong family should probably always select “latest” 😄

[dlech]

Plug the hub into the laptop with the USB cable

This must be the first step, otherwise the hub will turn off instead of reboot.

5\. Run a program with `print('reason:', hub.system.reset_reason())` though USB using `pybricksdev run usb`

The program must be run with BLE. USB is not supported at this time.

[TheVinhLuong102]

Today this bug has been elusive -- but definitely still lurking there. I only managed to catch this bug once, out of over a dozen tries, and I didn't manage to grab the reset_reason at that time. I don't know what has changed from my setup yesterday, when I caught it repeatedly. Maybe this thing is afraid of you David & Laurens. :D Or was there a solar flare yesterday? :)

I will keep trying again tomorrow.

Vinh

[TheVinhLuong102]

David, Laurens,

I've been testing a dozen more times again today. My observations are as follows:

1. When the hub is not plugged into a power source through USB, it would turn off after about a minute, sometimes earlier.

2. When it is plugged in, it does not fail like so.

So it seems there's something electrical about this. Does the Inventor Hub turn off when it is "overworked" in some sense?

Vinh

[dlech]

Is the battery low (flashing orange light)?

[TheVinhLuong102]

David, no, the battery was nearly full. To eliminate that risk, we charged it up before these tests.

[dlech]

Long button press, low battery and calling hub.system.shutdown() should be the only things that can intentionally make it turn off.

I wonder if there could be some sort of thermal shutdown going on. Does the hub get warm?

[TheVinhLuong102]

I wonder if there could be some sort of thermal shutdown going on. Does the hub get warm?

Not that we noticed. We'll retest again after a while and pay attention to that.

Also, let me also try substituting another hub in to see if this problem is hub-instance-specific.

[TheVinhLuong102]

Observed Experiment 1:

• Same Inventor Hub instance as before
• Hub battery fully charged
• Hug not plugged into power source
• Start running program through BLE: pybricksdev run ble --name "Pybricks Hub" remix/birthday-cake-cutter/main.py
• Leave the robot alone, doing nothing
• Hub is cool, not warm/hot
• Hub turns off after about 5 minutes

[laurensvalk]

I wonder if we're mixing two different things here.

All hubs turn off after 3 minutes if there is no connection and there is no program running. You're using pybricksdev, so it disconnects when the program ends or stops.

So just so we understand what you're saying, can you describe what the robot is doing prior to switching off? Is it idle or moving? What does the light display show?

Depending on your findings, the battery/usb/power aspect may be completely unrelated. If it turns off due to a freeze as described above, I wonder if it just turns back on since USB is connected.

[laurensvalk]

Can this be reproduced without building the robot, so with just the hub and a few motors and sensors attached?

[TheVinhLuong102]

Can this be reproduced without building the robot, so with just the hub and a few motors and sensors attached?

Laurens, yes we'll now attempt to do that. Now that the video on this robot is done, we'll dismantle it and try to reproduce this problem just with the connected motors.

I'll also substitute the other Prime Hub in to see if there's any such problems when we use a different hub.

[dlech]

FYI, we just fixed #304 which could be the same issue.

[TheVinhLuong102]

David, thank you. We'll try again to see the freezes/crashes with our current firmware (installed a few versions ago), and test whether the problem goes away with the latest firmware. We'll let you know the outcome.

[laurensvalk]

The beta installer hasn’t been updated yet, though. That gets updated when a new beta release comes out, just like for the other hubs.

If you want to try the latest build, you can use Pybricksdev, since you’ve got that installed anyway :)

[dlech]

Is this still a problem with v3.2.0b1?

[dlech]

I think I've had this happen to me recently a few times. Although it takes much longer than a minute before the hub turns off (nearly 30 minutes on the most recent occurrence). At first I dismissed it as possibly a low battery shutdown, but now I am starting the suspect there is something else going on.

I don't think I've run into this issue before because most of the time during development, I have the hub plugged in with USB.

[dlech]

On the other hand, the repeated crashes/sudden power offs I am currently seeing could be due to local changes in the firmware that I am experimenting with and could be unrelated to the original issue here.

[dlech]

Well, confirmed that at least the most recent crashes I saw were due to new code.

Perhaps the actionable item here is to figure out a way to get the watchdog timer do something else besides reset on the SPIKE Prime hub since the SPIKE Prime hub can't reboot without losing power.

[laurensvalk]

Since this was reported on (by now) fairly old firmware, I think we should close this. If unexpected power offs happen again, we can re-open.

[dlech]

This issue is that the hub can't keep power on during a watchdog timer reset. I see this every time we hit a crash on SPIKE Prime and occasionally on Technic hub. In theory, it is possible to get an interrupt from the watchdog timer before the reset and handle it some other way. For example, we could jump back to the reset vector instead of resetting the entire MCU. But we would need to figure out which IP blocks need to be reset to ensure all of the hardware keeps working properly.

[balisco1]

Good day, what I figured out here is the problem is the battery contact, try to fill the space in the battery contact to be tighter so that it will not shake

[emilebosman39154]

Let me reinstall the firmware -- both official and Pybricks -- and see if this problem remains.

Hallo Vinh, hope you can help please. Where do I find the latest official downloadable (FTP) firmware for the Lego Spike Prime? Also is there a document to indicate how to reinstall the firmware - We do not want to cause a "lockdown" through our ignorance. Thank you. Kind regards Emile

[laurensvalk]

To restore the official firmware, go to https://code.pybricks.com, go to the settings tab, and click 'restore official firmware'.

Follow the steps for your hub.

After that, connect to the respective LEGO app and you'll automatically get the latest firmware for your hub.