PiJuice won't reboot my pi on rare occasions

[blanco-ether]

This issue is not identical to: PiJuice won't reboot my pi on rare occasions

My PiJuice application does however incorporate a Pi in a box with official touchscreen. I use SW1 connected to an external button for power on/power off. A user does a SINGLE_PRESS to power the Pi into a python GUI application for programming circuit boards. When they are done with a board run they do a LONG_PRESS1 to power down. Works fine 99% of the time.

The run pin is not installed. I have multiple boxes, 2 - RPi3, 2 - RPi4, they all exhibit the same problem occasionally.

The configuration for SW1 is:

PRESS: NO_FUNC, 0 RELEASE: NO_FUNC, 0 SINGLE_PRESS: HARD_FUNC_POWER_ON, 800 DOUBLE_PRESS: NO_FUNC, 0 LONG_PRESS1: SYS_FUNC_HALT_POW_OFF, 1000 LONG_PRESS2: HARD_FUNC_POWER_OFF, 10000

The boxes are most often used by individuals other than myself. I've tried everything to reproduce this on my own and haven't been able to.

Any ideas?

Thanks

[shawaj]

@blanco-ether what battery are you using?

[blanco-ether]

I am using a 5000mah LiPo that does not have temp sensor

[shawaj]

@blanco-ether so it is your own battery? Do you have a link to it? And what battery profile are you using?

[blanco-ether]

Battery is 3.7V 5000mAh 115659

using PJLIPO_5000 profile.

[shawaj]

Where did you buy it from?

Also is it a PiJuice HAT or Zero?

[blanco-ether]

Amazon

PiJuice HAT

Is difficult to find official PiJuice batteries in US.

[blanco-ether]

Forgot to mention that remedy is always remove/restore battery connection effectively resetting the PiJuice.

The battery appears to have integral temp sensor/current limit within clear wrapper.

[shawaj]

@mmilann do you think this could be some firmware bug causing it to get stuck in a weird state?

@blanco-ether is the problem only when you are trying to turn it off? It always works when trying to turn it on?

[blanco-ether]

Problem is turning it on. Turning it off always works.

When PiJuice is unable to turn the Pi back on, LED activity appears identical to when everything is working fine in charge/no charge states. Blue when charging, flashing green when not. I do not have one in stuck state right now for reference but I can report that when is stuck and I press SW1 remote button I see brief solid on LED activity on the PiJuice but nothing happens on the RPi. Also, no LEDs on Pi are on when in stuck state. So I conclude that Pi is definitely powered down.

Other info is we are using firmware v1.4 on all PiJuice now. We did see the problem on v1.3 also. I will update to v1.5 asap. We have seen the issue for for quite some time and have been treating it as a nuisance. A problem though with briefly disconnecting battery though is that we lose RTC and our board programming/test fixture logs results and time is no longer correct in logs.

I did have pogo pins installed at one time and saw the problem in that case also. I removed the pins in hope that may cause the problem. We don't require any timed startup anyway. Only startup/shutdown by user through SW1.

Thank you for investigating.

[blanco-ether]

Other info I can offer is that in our use case we seek to protect against abrupt power loss in every possible way.

No battery turn on is not enabled.

We safely assume that battery is always charged. This is so because our application monitors no power and if not present displays a countdown warning user of imminent shutdown if user does not plug in. We also use No power system event to run USER_FUNC1 which is a python script that also monitors and does shutdown at a time later than would occur if App were running and handled the no power condition. This is to safeguard against prolonged running on battery power if our App is not running which would be the case during development. Both of these safe guards work perfect. In short, a user can only effectively use the RPi in a box if external power is applied and our setup only tolerates running on battery with no power for less than a minute at a time. Battery level always indicates >90%.

We experienced failure to power on prior to adding these safeguards in. I only mention them to help convey everything our setup uses.

[mmilann]

There is watchdog that resets mcu if it gets stucked with 8s period.

@blanco-ether How power is connected to PiJuice or Rpi? After what time you retried turning on, because there is 15 sec of dead time between wake-up retries forr stability reasons?

[blanco-ether]

Power is applied to J4 on PiJuice. Power supply is 5V, 5A with 3A fuse.

When power on fails, Pi in a box typically has been dormant for days or weeks. It is not typical that box is powered then powered down then back on again back to back other than by myself in development for testing. Even in this case I cannot reproduce stuck state that users experience.

When fail to power on happens I will investigate users report asap and I confirm what they see and am at a loss to explain. We cycle battery power to be able to power on again. They assure me that they simply single press to power on, long press to power off. After some period of dormancy they report they can't power up using single press.

[tvoverbeek]

What is the parameter of the single press action? It is in millisec. When it is e.g. 800 then if you keep the button pressed longer than .8 sec the action will not be executed.

[blanco-ether]

Yes, it is 800.

If the button is pressed greater than .8s would it cause the Pi to not power on again?

When I attend to a box that is no longer able to power the Pi on again I single press it it myself (correctly less than .8s) and does not power.

[mmilann]

Because you used SYS_FUNC_HALT_POW_OFF, is Rpi off in that condition, Rpi LED is off or on?

[blanco-ether]

I make the assumption that when long press applied (SYS_FUNC_HALT_POW_OFF), the Pi is fully booted and thus red LED on. I can try long press when not booted and see what happens. I don't think users do that by accident anyway.

Red LED does always go off after about 30s when shut down.

[blanco-ether]

I have never seen a case where red LED on Pi was on when fail to boot condition exists.

I just tried combinations of long press when is off. In all cases when I let button up then apply single press it boots.

The problem most often occurs when others are using the box and I am called for help and investigate. It has only happened to me twice. Once early on in development and I dismissed it as a fluke. Another time was after users reported the condition several times. In that case I did nothing out of the ordinary. It also validated that users were unlikely doing anything out of the ordinary either.

Thanks again for looking into this.

[mmilann]

Most clear condition is when Rpi is off (GPIO 5V off) and system powered over PiJuice power input (J4 or USB), then after single press pijuice just turns 5V GPIO power on (supposing it is not un-powered, bad battery).

[blanco-ether]

Hi mmilann, Not sure if you are asking a question or stating what should happen.

Next time condition exists I can ask the user to leave the box as is. I can take a reading on 5v GPIO and report it to you.

When no start condition exists, PiJuice by all appearances (LED activity) is running fine. It even registers LED activity when SW1 pressed. Pi simply does not start.

[mmilann]

@blanco-ether trying to catch you scenario on my setup and it would be useful if you can obtain conditions like Rpi GPIO 5V status.

Pijuice measures 5V at GPIO and in case it is on power rest is initiated (GPIO 5v off then on), if it is off then it is just turned on. There are some conditions prior wake-up like if only powered from battery, battery voltage must be over threshold (3V).

[blanco-ether]

I have asked the primary user at our location that if RPi fails to start to notify me so that I can take readings on 5v GPIO and battery. Anything else I should look for or readings I should take? We have resources to take any measurements that might be helpful. I can even record video to show LEDs etc if that would be helpful. It may be a while before the condition shows up but I can ask them to try to start it up every day just to increase chances of happening.

Another possible clue I can offer is that we are using official Pi touchscreen. Touchscreen power is connected to 5v GPIO on the Pi. Other than touchscreen ribbon cable connected to the Pi we do not have any other connections to the touchscreen. No USB power or anything else is connected to the touchscreen.

In the past when Pi failed to start I did disconnect/reconnect external power and try to start. In either case it did not help.

[mmilann]

@blanco-ether any observation about Rpi state when it failed to wake-up would be useful, like is there any sign of GPIO voltage gets power for short period Rpi LED may blink.

I can suspect that official touchscreen makes overload on power-up, like making over-current condition because of charging large capacitor.

[blanco-ether]

Today I attempted to turn on 2 boxes and was not able to demonstrate the problem. I had a conversation with a user about not able to turn on the RPi. He was using it at the time (already powered up). He said lets try it now. He powered down the box and attempted to restart. It did not. I saw it with my own eyes! I measured battery on PiJuice battery terminal and it reads 4.178v. Reading on 5v GPIO on the RPi is 0.096v. When I press the button (SW1) and read 5v GPIO I do not see any change. When no external power applied I see green LED flash about once a second. Again with no external power, When button (SW1) pressed the LED goes on solid for about 1s so I can see button press is registered. When external power is applied same LED is solid blue and lights green about once a second. When SW1 pressed in this case it also seems to go green for about 1s registering keypress. Red LED on RPi is OFF at all times. No sign of any activity on RPi.

I will point out that when the user did this in front of me he attempted to restart before RPi red LED was out. This is NOT typical. When RPi fails to start it usually has been dormant for days or weeks.

I will leave the PiJuice and RPi in the stuck state in case you would like me to take any more readings. I have also taken some video of LED activity on PiJuice that verifies what I have reported. If you would like to see let me know.

The box that is stuck now is RPi4 and uses LIPO battery PL706090 3.7v, 5000mah. It has integral temp/current limit control.

[mmilann]

@blanco-ether Thanks for reporting! I tried to repeat similar conditions as Rpi touchscreen by attaching 8R2 resistor load (load current of ~600mA) plus capacitor of 1000uF at 5V GPIO. This produces faulty state (on power button press) that I suspect may be your case, beacuse touchscreen may have large capacitance on power rail >= 470uF. 5V DCDC gets overloaded (current limit protection at turn on), having large capacitive load plus total start current of Rpi and load resistor over 1A. This faulty state manifests as near 0V at 5V GPIO but with short pulses every 20ms where DCDC retries to turn on, however it continuously re-enetrs into faulty state. If possible, can you use oscilloscope to observe signal at 5V GPIO to confirm is there similar behaviour in your case pulsed signal with 20ms period?

[blanco-ether]

I took a reading and signal almost appears to be noise however does have a period of about 20ms.

Thanks for looking into this!!

[mmilann]

@blanco-ether good to hear that we are approaching problem. Will work on firmware patch for this situation and if possible take photo of oscilloscope to compare.

[blanco-ether]

I was thinking after I left the office that I could disconnect the touchscreen power from GPIO and then try to boot. If it did boot the Pi would that prove the case? I will leave it in non-powering state until I hear back in case you need other readings.

[mmilann]

One case is that power is connected to pijuice USB micro, another no power connected and powering from battery. Load or touchscreen and Rpi are powered from 5V GPIO sourced by pijuice. Faulty state may happen on power-up or wake-up, like button press, where 5V DCDC goes to faulty state, so Rpi and touchscreen will not be powered but there will be periodic train of pulses on 5V GPIO (20ms period). If you got in state where it fails to power-up Rpi and touchscreen you can measure 5V GPIO with oscilloscope and post signal so I can confirm if it is same type of issue.

[blanco-ether]

About what voltage level are the pulses you see? I will post oscilloscope screen shot later today.

[mmilann]

Pulses are around 2V-3V amplitude.

[blanco-ether]

I do not have the same here. Level is in the mv range.

Our problem may not be the same.

Another thing that occurred to me is that we always have a bluetooth dongle plugged into Pi USB to communicate to a barcode scanner. Barcode scanner is plugged into 5v power, not RPi.

I haven't cycled battery power yet so Pi/PiJuice is still in stuck state.

[mmilann]

It does not look like DCDC overload fault retries, however possible to be dcdc disabled. If anyway this is same kind of issue there will be fix in firmware V1.5.

Further thing you can test is that in fault state get access to pijuice status, pijuice_cli or pijuice_gui. You can directly plug power to Rpi in order power it up while pijuice is in same fault state and then run pijuice_cli or gui to see what status says.

[tvoverbeek]

@mmilann Is this fix already in 1.5? The wakeup-on-charge Restore option?

[mmilann]

@tvoverbeek no, DCDC overload fix will be pushed as new v1.5 update.

[blanco-ether]

@mmilann Thanks or the notice. I will try it as soon as is available.

Am trying to track down USB-C power cable to power the Pi to check pijuice-cli.

[blanco-ether]

@mmilann I powered the Pi directly from USB-C. My safeguards caused the Pi to shut down as I did not have power applied to PiJuice J4 also. As such I was not able to get into pijuice-cli. After powering Pi directly through USB-C I am now able to start up through SW1. Don't know what that tells us.

[mmilann]

@blanco-ether potentially could be DCDC fault because power from USB-C pre-charges capacitors and helps turn-on. It would be interesting to see if pijuice responds over I2C in stucked state and what show reading in status window.

[blanco-ether]

@mmilann If RPi gets stuck again I will attempt USB power and try pijuice_cli

[mmilann]

@blanco-ether firmware update released: https://github.com/PiSupply/PiJuice/blob/v1.7/Firmware/PiJuice-V1.5_2021_02_04.elf.binary I recommend you to test it. there are several fixes that may address you issue. Yo can also install it with new software package (currently in test phase), https://github.com/PiSupply/PiJuice/blob/v1.7/Software/Install/pijuice-base_1.7_all.deb

[blanco-ether]

@mmilann Very Good!

I will install it asap and report results.

Thanks

[blanco-ether]

@mmilann Link to firmware is no longer active, I can't update the rest of our RPi to test the fix.

[mmilann]

@there was update, new, link for software base package: https://github.com/PiSupply/PiJuice/blob/v1.7/Software/Install/pijuice-base_1.7_all.deb

or just firmware if you have already installed firmware V1.5 but older releases: https://github.com/PiSupply/PiJuice/blob/v1.7/Firmware/PiJuice-V1.5_2021_02_06.elf.binary

[blanco-ether]

@mmilann Thanks. Installed on 4 RPis. Will report back in time after some testing.

[blanco-ether]

Been testing power on/off several times a day for

[blanco-ether]

Been testing power on with 3 RPi's several times a day for last 4 days and so far so good!

Will continue to report results.

On another note, I am wondering about the connector you use to connect JTag to your board when programming in production. Am curious because I would like to use something like it on our own circuit boards. It appears as though the connector is aligned with holes on PiJuice circuit board and has pogo pins. Where do you get it?

Thanks for all your help!!

[mmilann]

@blanco-ether good to hear about new firmware testing status.

Take a look at TC2050-IDC.

[blanco-ether]

Just want to report that we have not seen the startup problem since firmware update. We have been testing by turning on 3-4 RPis several times a day and so far looks like the problem has been fixed. One thing I can note is that it seems to take about 4s longer for RPi 4 to show activity on display when start button pressed. Maybe because of higher current draw. RPi 3 screen lights about 1.5s after pressing start. This is only an observation not a problem although sometimes it looks like RPi 4 is not going to start.

Thank you for fixing!!

[mmilann]

Great, thanks for report!