WeatherChimes devices crash after variable length of time

[WL-Richards]

Describe the bug Devices have been shown to crash after what could range from a few days to a week

Hardware in Use Standard WeatherChimes hardware (Tipping Bucket)

Expected behavior These device should not be crashing

Code

/**
 * In lab use case example for the WeatherChimes project
 * 
 * This project uses SDI12, TSL2591 and an SHT31 sensor to log environment data and logs it to both the SD card and also MQTT/MongoDB
 * 
 * MANAGER MUST BE INCLUDED FIRST IN ALL CODE
 */
#include <Loom_Manager.h>
#include <Logger.h>

#include <Hardware/Loom_Hypnos/Loom_Hypnos.h>

#include <Sensors/Loom_Analog/Loom_Analog.h>
#include <Sensors/I2C/Loom_SHT31/Loom_SHT31.h>
#include <Sensors/I2C/Loom_TSL2591/Loom_TSL2591.h>
#include <Sensors/I2C/Loom_MS5803/Loom_MS5803.h>

#include <Internet/Logging/Loom_MQTT/Loom_MQTT.h>
#include <Internet/Connectivity/Loom_LTE/Loom_LTE.h>

// Pin to have the secondary interrupt triggered from
#define INT_PIN A0

volatile bool sampleFlag = true; // Sample flag set to 1 so we sample in the first cycle, set to 1 in the ISR, set ot 0 end of sample loop
volatile bool tipFlag = false;
volatile int counter = 0;

// Used to track timing for debounce
unsigned long tip_time = 0;
unsigned long last_tip_time = 0;

Manager manager("Chime", 3);

// Create a new Hypnos object
Loom_Hypnos hypnos(manager, HYPNOS_VERSION::V3_3, TIME_ZONE::PST, true);

// Analog for reading battery voltage
Loom_Analog analog(manager);

// Create sensor classes
Loom_SHT31 sht(manager);
Loom_TSL2591 tsl(manager);
Loom_MS5803 ms_water(manager, 119); // 119(0x77) if CSB=LOW external, 118(0x76) if CSB=HIGH on WC PCB
Loom_MS5803 ms_air(manager, 118); // 118(0x76) if CSB=HIGH on WC PCB

Loom_LTE lte(manager, "hologram", "", "");
Loom_MQTT mqtt(manager, lte.getClient());

/* Calculate the water height based on the difference of pressures*/
float calculateWaterHeight(){
  // ((Water Pressure - Air Pressure) * 100 (conversion to pascals)) / (Water Density * Gravity)
  return (((ms_water.getPressure()-ms_air.getPressure()) * 100) / (997.77 * 9.81));
}

// Called when the interrupt is triggered 
void isrTrigger(){
  sampleFlag = true;
  detachInterrupt(INT_PIN);
  hypnos.wakeup();
}

void tipTrigger() {
  hypnos.shouldPowerUp = false;
  tipFlag = true;
  detachInterrupt(INT_PIN);
}

void setup() {

  // Enable debug SD logging and function summaires
  ENABLE_SD_LOGGING;
  ENABLE_FUNC_SUMMARIES;

  // Set the interrupt pin to pullup
  pinMode(INT_PIN, INPUT_PULLUP);

  // Wait 20 seconds for the serial console to open
  manager.beginSerial();

  // Enable the hypnos rails
  hypnos.enable();

  // Read the MQTT creds file to supply the device with MQTT credentials
  mqtt.loadConfigFromJSON(hypnos.readFile("mqtt_creds.json"));

  // Initialize all in-use modules
  manager.initialize();

  // Register the ISR and attach to the interrupt
  hypnos.registerInterrupt(isrTrigger);

  attachInterrupt(INT_PIN, tipTrigger, FALLING);
}

void loop() {

  if(sampleFlag){
    // Measure and package the data
    manager.measure();
    manager.package();

    manager.addData("Tip_Bucket", "Tip_Count", counter);

    // Add the water height calculation to the data
    manager.addData("Water", "Height_(m)", calculateWaterHeight());

    // Print the current JSON packet
    manager.display_data();            

    // Log the data to the SD card              
    hypnos.logToSD();

    // Publish the collected data to MQTT
    mqtt.publish();

    // Set the RTC interrupt alarm to wake the device in 15 min
    hypnos.setInterruptDuration(TimeSpan(0, 0, 15, 0));

    // Reattach to the interrupt after we have set the alarm so we can have repeat triggers
    hypnos.reattachRTCInterrupt();
    attachInterrupt(INT_PIN, tipTrigger, FALLING);
    attachInterrupt(INT_PIN, tipTrigger, FALLING);
    sampleFlag = false;
  }

  if(tipFlag){
    digitalWrite(LED_BUILTIN, HIGH);
    delay(20);
    counter++;
    tipFlag = false;
    attachInterrupt(INT_PIN, tipTrigger, FALLING);
    attachInterrupt(INT_PIN, tipTrigger, FALLING);
    digitalWrite(LED_BUILTIN, LOW);
  }

  // Put the device into a deep sleep, operation HALTS here until the interrupt is triggered
  hypnos.sleep();

}

EDIT 8/11/2023 - Updated code to latest version where issue is still present

Additional context Mainly opening this issue just show I have a place to log why it was crashing, device just crashed and so I will look at where it crashed by pulling the SD card when I am in the lab next.

[WL-Richards]

Serial log showed that the device crashed before the first print of the next cycle

[WL-Richards]

Upon further review, it appears that the power_up call to one of the sensors is initiating the hang

[WL-Richards]

Unsure what the problem is currently, in the interest of the chimes working in Alaska I moved the watchdog timer to turn on as soon as the device exits sleep mode to prevent hangs, currently testing

[WL-Richards]

Issue #67 may have something to do with this

Getting better log output to determine cause of failure

[WL-Richards]

I'm almost certain this was caused by the strings memory leakage issue that plagued all of loom, I need to run duration tests to be sure

[WL-Richards]

Nope the strings overhaul seemed to fix most other devices but the weatherchimes hangs in what appears to be the SD card initialization

[WL-Richards]

Bump, we have narrowed it down to the likely culprit being the additional interrupt for the tipping bucket causing the device to hang. Still unsure why

[WL-Richards]

At the current moment in time it appears as though standard WeatherChimes functionality is stable on main, closing.... finally