Particle Cloud Disconnection

[sampathsatti]

Issue

Particle devices are getting disconnected from the cloud. Currently both devices at JHS and one device at REACH (REACH_1) are currently disconnected.

Relevant staff have been informed of the outage and appointments to go in and inspect the devices have been made

JHS - Friday 3:00PM REACH - Tuesday 9:00AM

Previous experience and obeservations

These obeservations have been recorded on Particle Photon devices which are markedly different from the Argon Gen 3 device.

This issue has most frequently been observed at JHS. The heartbeat alert was triggered and the particle device could not be reached through the particle cloud.

During the site visit, pressing the reset/setup buttons seemed to have no effect. The one fix that enabled the system to reconnecte to the cloud and startup was plugging the USB sensor out and back in again. No voltage measurements were performed on the pins at that time.

I hypothesized that this was probably because of the stripped solder mask on the XeThru causing a short between VCC and GND affecting the 3V3 voltage output. Ironically, any damage due to this hypothetical failure mode was reversible.

The repeat disconnection events happened on Brand new Radar modules with Plastic screws holding everything together.

Implications

• Supporting remote locations would be quite hard if this failure mode exists on Argon Devices as well
• Lack of visibility into why this is happening is concerning
• Associate Rework is expensive and takes dev time away from dev to ops/maintainence

Photon Hardware

Photon Schematics

Power module

Converts VUSB to 3.3V

• Takes VIN (VUSB minus Schottky Diode forward voltage drop) Expected to be close to 4.8V
• XeThru is powered by VIN
• Enable is always pulled high by connecting it to VIN through 100k resistor

P0 WiFi Module

• All the particle functionality
• Is powered by 3.3V
• WKP is left unconnected
• nRESET (Usually held HIGH => Don't reset)

Argon Hardware

Argon Schematics

XC9528A

3.3V Voltage Regulator

• Takes VIN (VUSB minus Schottky Diode forward voltage drop)
• Output is 3.3V = VDD and V_WiFi
• These two are separated into breanches with inline inductors to deliver local current demands and make them independent
• Enable pin that makes this chip work or not (and the rest of the device actually)... If enable is low, it may not run. Is enable being pulled low?

XC6802

LiPo Charger

• Not at play in this particular scenario

nRF52840

Responsible for the BLE aspect - and is microcontroller

• Takes VDD as input power (3.3V)
• It has its own flash memory
• Also connects to VBUS and the D+, D- for programming

ESP32

Handles WiFi

• Takes V_WiFi as input power (3.3V)
• Manages its own flash
• Only way to interface with it is through the nRF52840

Plan of action for next site visit

Discussion w/ Heidi

• Suggestion by Heidi "Check voltages of all the pins"
  • If board is receiving VIN, is it outputting VCC (3.3V?)
  • What is nRST at?
  • What is WKP at?
• Count the number of times the device is connecting/disconnecting from network
  • Store in EEPROM
  • If it hits a certain count, trigger a hard reset of the device

Discussion w/ Dave

• Include hardware sensors for mitigation (temperature)
• Check logs (Software debug)
• Electrical debug

Sampath's thoughts

• Knowing more about what happened is good, but is limited in utility if we don't have the ability to act on it.
• Are the disconnection events caused by the same reason at each site? Are the symptoms the same? Are they similar to what we have seen before?
  • If same as previous
• We shouldn't turn off the Particle's Cyan breathing - I think it can be useful field debugging tool

Hotfixes

• On the particle argon - Connect a push button the enable pin and customers get notified asking them to press the button within the heartbeat text message

• On the particle photon - The PCB contains a micro USB adapter which is plugged in. There is a current sensor in-linbe measuring the current drawn by the Particle. On the current being 0, or on a particle command, an electrical relay toggles the power effectively replicating the behavior of unplugging/plugging back in

[sampathsatti]

Notes from JHS Site visit

Observations

Voltage Measurement

The same voltage measurements were seen on both devices. Voltages were as expected

Trace	Voltage in V
Vcc	4.68
3V3	3.36
nRST	3.36
WKP	0.95

Interventions

RESET

• Pressing the reset button caused the LEDs to flash momentarily
• The voltmeter was measuring the voltage of the RST pin as the button was pressed
• Voltage became 0 when button was pressed as expected
• Devices reconnected to the cloud

WiFi Network change

• Photons were reconnected to JHS_Fraser_EXT, a network created by the TP-Link WiFi Range Extender.

Conclusions and recommendations

• Unlikely that this was a hardware issue
• Previously when the photons disconnected, simply pressing the reset switch was not enough to bring them back online.
• Different reason at play - More firmware diagnostics may be required to find out more details
• Do not turn off Particle Status LED - Information is lost in the process. It does not draw attention to the sensor since the LED faces upwards...
• Site visit could have been avoided by simply drawing out a push button to the enclosure and asking client to press it to bring system back online.
• Verify if the issue at REACH is similar

[sampathsatti]

Failure modes and effects analysis

Failure modes and effects analysis (FMEA) was conducted to understand this cloud disconnection failure mode better and identify mitigation steps. Link here

• As a result of this failure mode, false negatives could occur (an adverse health event happening in the washroom that goes undetected/does not result in an alert. The severity of an even like this is rated at 10.
• In addition, this failure mode requires associate rework to correct and set right.
• The event has been happening quite frequently, possibly multiple instances in a month. Frequency score is 8.
• The only available controls to identify this failure mode were manually looking through the dashboard or the particle console. This resulted in a detectability score of 10.

These scores resulted in a Risk Potential Number (RPN) of 720, requiring immediate mitigation steps.

Some of the mitigation steps identified were

• Device reset w/o associate rework
• Heartbeat alerts to be reimplemented
  • Have alerts coming from a separate number so as to separate out noise from signal
• Have a debugging checklist
• List Argon-accessible Failure mode indicators
• Hardware and Firmware changes to expose indicators