MQTT with batchSD hangs code if batch size is too large; also counts packets incorrectly

[solidstatebird]

Describe the bug In Loom a079c45, MQTT upload will hang the program execution if the current batch is too large. Additionally, MQTT counts the packets in a batch incorrectly, counting one packet past the limit, which is not transmitted with the next batch.

Hardware in Use Feather m0 with ATWINC1500 plugged into Hypnos v3.3 with SD card.

To Reproduce Steps to reproduce the behavior:

1. Set BATCH_SIZE to
2. Upload code to Feather m0 with ATWINC1500 using loom
3. Observe serial output

Expected behavior All packets should be uploaded to the database, without any omissions or program crashes.

Code This code generates packets with dummy data and attempts to upload them in batches.

#include <Loom_Manager.h>
#include <Hardware/Loom_Hypnos/Loom_Hypnos.h>
#include <Internet/Connectivity/Loom_Wifi/Loom_Wifi.h>
#include <Internet/Logging/Loom_MQTT/Loom_MQTT.h>

#define BATCH_SIZE 6

Manager manager("mqtt_bug_test", 99);
Loom_Hypnos hypnos(manager, HYPNOS_VERSION::V3_3, TIME_ZONE::PST);

#include "arduino_secrets.h"
Loom_WIFI wifi(manager, CommunicationMode::CLIENT, SECRET_SSID, SECRET_PASS);
Loom_MQTT mqtt(manager, wifi.getClient(), SECRET_BROKER, SECRET_PORT, MQTT_DATABASE, BROKER_USER, BROKER_PASS);
Loom_BatchSD batchSD(hypnos, BATCH_SIZE);

void setup()
{
    manager.beginSerial(true); 

    hypnos.enable();
    manager.initialize();
}

void loop()
{
    manager.measure();
    manager.package();
    for(int i = 0; i < 10; i++) {
      manager.addData("Name" + String(i), "DummyData" + String(i), "DummyData" + String(i));
    }
    hypnos.logToSD();

    mqtt.publish(batchSD);

    delay(2500);
}

arduino_secrets.h:

#define SECRET_SSID "hotspot"
#define SECRET_PASS "aaaabbbb"

#define MQTT_DATABASE "MQTT_BUG_TEST"

#define SECRET_BROKER ""
#define SECRET_PORT 
#define BROKER_USER ""
#define BROKER_PASS ""

Output Batch size (6) is too large (code crash)

[SD Manager] Initializing SD Card...
[SD Manager] Successfully initialized SD Card!
[SD Manager] Data will be logged to mqtt_bug_test10.csv
[Hypnos] DS3231 Real-Time Clock Initialized Successfully!
[Manager] ** Initializing Modules **
[WiFi] Initializing WiFi module...
[WiFi] Attempting to connect to SSID: hotspot
[WiFi] Attempting to connect to AP...
[WiFi] Connected to network!
[WiFi] Verifying Connection to the Internet...
[WiFi] Successfully Pinged Google! Response Time: 100ms
[WiFi] Successfully Initalized Wifi!
[WiFi] Device IP Address: (omitted)
[WiFi] Device Subnet Address: 255.255.255.0
[Manager] ** Setup Complete ** 
[SD Manager] Successfully logged data to mqtt_bug_test10.csv
[MQTT] Creating new MQTT client!
[MQTT] Batch not ready to publish: 1/6
[SD Manager] Successfully logged data to mqtt_bug_test10.csv
[MQTT] Batch not ready to publish: 2/6
[SD Manager] Successfully logged data to mqtt_bug_test10.csv
[MQTT] Batch not ready to publish: 3/6
[SD Manager] Successfully logged data to mqtt_bug_test10.csv
[MQTT] Batch not ready to publish: 4/6
[SD Manager] Successfully logged data to mqtt_bug_test10.csv
[MQTT] Batch not ready to publish: 5/6
[SD Manager] Successfully logged data to mqtt_bug_test10.csv
[MQTT] Attempting to connect to broker: cas-mosquitto.biossys.oregonstate.edu:1883
[MQTT] Successfully connected to broker!
[MQTT] Attempting to send data...

the code has crashed at this point

Batch size (3) is sufficiently small (packet miscount)

[SD Manager] Initializing SD Card...
[SD Manager] Successfully initialized SD Card!
[SD Manager] Data will be logged to mqtt_bug_test8.csv
[Hypnos] DS3231 Real-Time Clock Initialized Successfully!
[Manager] ** Initializing Modules **
[WiFi] Initializing WiFi module...
[WiFi] Attempting to connect to SSID: hotspot
[WiFi] Attempting to connect to AP...
[WiFi] Connected to network!
[WiFi] Verifying Connection to the Internet...
[WiFi] Successfully Pinged Google! Response Time: 90ms
[WiFi] Successfully Initalized Wifi!
[WiFi] Device IP Address: (omitted)
[WiFi] Device Subnet Address: 255.255.255.0
[Manager] ** Setup Complete ** 
[SD Manager] Successfully logged data to mqtt_bug_test8.csv
[MQTT] Creating new MQTT client!
[MQTT] Batch not ready to publish: 1/3
[SD Manager] Successfully logged data to mqtt_bug_test8.csv
[MQTT] Batch not ready to publish: 2/3
[SD Manager] Successfully logged data to mqtt_bug_test8.csv
[MQTT] Attempting to connect to broker: cas-mosquitto.biossys.oregonstate.edu:1883
[MQTT] Successfully connected to broker!
[MQTT] Attempting to send data...
[MQTT] Publishing Packet 1 of 3
[MQTT] Publishing Packet 2 of 3
[MQTT] Publishing Packet 3 of 3
[MQTT] Data has been successfully sent!
[SD Manager] Successfully logged data to mqtt_bug_test8.csv
[MQTT] Batch not ready to publish: 4/3
[SD Manager] Successfully logged data to mqtt_bug_test8.csv
[MQTT] Batch not ready to publish: 1/3
[SD Manager] Successfully logged data to mqtt_bug_test8.csv
[MQTT] Batch not ready to publish: 2/3
[SD Manager] Successfully logged data to mqtt_bug_test8.csv
[MQTT] Successfully connected to broker!
[MQTT] Attempting to send data...
[MQTT] Publishing Packet 1 of 3
[MQTT] Publishing Packet 2 of 3
[MQTT] Publishing Packet 3 of 3
[MQTT] Data has been successfully sent!
[SD Manager] Successfully logged data to mqtt_bug_test8.csv
[MQTT] Batch not ready to publish: 4/3
[SD Manager] Successfully logged data to mqtt_bug_test8.csv
[MQTT] Batch not ready to publish: 1/3
[SD Manager] Successfully logged data to mqtt_bug_test8.csv
[MQTT] Batch not ready to publish: 2/3
[SD Manager] Successfully logged data to mqtt_bug_test8.csv
[MQTT] Successfully connected to broker!
[MQTT] Attempting to send data...
[MQTT] Publishing Packet 1 of 3
[MQTT] Publishing Packet 2 of 3
[MQTT] Publishing Packet 3 of 3
[MQTT] Data has been successfully sent!

[WL-Richards]

Packet miscount issue fixed in 297dc9b

[WL-Richards]

As for the hanging behavior it appears to never return from this line: Loom_BatchSD:19 String fileOutput = sdMan->readFile(sdMan->getBatchFilename());

[WL-Richards]

This has all been fixed in the string-overhaul branch