issuse

[b1n0-kun]

Hi Mobitz, it's me again.

This time, I want to ask about the Backup node into Flash memory. When I backup with this line of code (from your example):

if (Firebase.backup(fbdo, StorageType::FLASH, databasePath +"/readings" / node path to backup/, "data/FB_backup/backup.txt" / file name included path to save /, rtdbDownloadCallback / callback function /)) { Serial.printf("backup file, %s\n", fbdo.getBackupFilename().c_str()); Serial.printf("file size, %d\n", fbdo.getBackupFileSize()); } else Serial.println(fbdo.fileTransferError().c_str());

The result is this error printed in the Serial

-> file is still opened

I am using BrowersFS to explore the file in LittleFS, the "backup.txt" file is created by ESP but can't write the content into it

Tried to google but couldn't find anything similar. Can you help me on this ?

Arduino IDE version: 1.8.16

ESP8266 Arduino Core SDK version

• Version 4.3.11

[mobizt]

Thanks for reporting this issue.

To fix this issue, please update the library to v4.3.14.

[b1n0-kun]

Thank you. I'm waiting for the new patche comes online. Will feedback to you when I finish.

[b1n0-kun]

Hi Mobitz. After I update the library to 4.3.14, the problem is still there. To be more precise, after 1st attempt of backup, it said -> Download failed, and then from 2nd attempt, it return -> file is still opened. Is there any specific requirement for the backup file ?

[mobizt]

it's not possible.

Uninstall library by delete from libraries folder and install the latest version from Library Manager.

[b1n0-kun]

I tried that delete and re-installed library also, the issue still there. Even after I manually deleted the backup file created by esp and tried to backup again, it still showed "file is still opened"

[mobizt]

In ESP8266, you should reserve the BearSSL rx buffer large enough for your data at that node to download and upload.

As this is the test result of node data size up to 10k bytes when download.

The BearSSL rx buffer size should be 10k up too (12k in this case) as following.

fbdo.setBSSLBufferSize(12000/* Rx buffer size in bytes from 512 - 16384 */, 512 /* Tx buffer size in bytes from 512 - 16384 */);

Test result of 10k data backup (download) and restore (upload)..

Connecting to Wi-Fi...........
Connected with IP: 192.168.137.30

Firebase Client v4.3.14

Token info: type = id token (GITKit token), status = on request
Token info: type = id token (GITKit token), status = ready
Downloaded 10%
Downloaded 19%
Downloaded 29%
Downloaded 39%
Downloaded 49%
Downloaded 59%
Downloaded 69%
Downloaded 79%
Downloaded 89%
Downloaded 99%
Downloaded 100%
Backup completed

backup file, /test.bak
file size, 10312

Restore...

Uploading file /test.bak (10312) to /test_backup
Uploaded 4%
Uploaded 9%
Uploaded 14%
Uploaded 19%
Uploaded 24%
Uploaded 29%
Uploaded 34%
Uploaded 39%
Uploaded 44%
Uploaded 49%
Uploaded 54%
Uploaded 59%
Uploaded 64%
Uploaded 69%
Uploaded 74%
Uploaded 79%
Uploaded 84%
Uploaded 89%
Uploaded 94%
Uploaded 99%
Uploaded 100%
Restore completed

[mobizt]

If the BearSSL rx buffer is too small to handle the data, the invalid or truncated SSL record will be stored in rx buffer and cannot decrypt it to raw (text) data then no or empty data returns from BearSSL engine.

[b1n0-kun]

I raised the BearSSL to 12000 as you did, and successfully backup, but only with maximum of 2 child nodes, when I raised to 3 child node, Download failed, and the maximum of the backup file I got here is 294B

Kinda surprise with your test succesfully with file size up to 10kb...

Here is content of my child node, nothing too heavy apparently

[mobizt]

It's normal that library is currently supported large file download/upload (and OTA) in RTDB.

Megabytes of data download/upload tested.

You can test with this complete test code.

#include <Arduino.h>
#include <ESP8266WiFi.h>
#include <FirebaseESP8266.h>

#include <addons/TokenHelper.h>
#include <addons/RTDBHelper.h>

#define WIFI_SSID "WIFI_AP"
#define WIFI_PASSWORD "WIFI_PASSWORD"

#define API_KEY "API_KEY"

#define DATABASE_URL "URL"

#define USER_EMAIL "USER_EMAIL"
#define USER_PASSWORD "USER_PASSWORD"

FirebaseData fbdo;

FirebaseAuth auth;
FirebaseConfig config;

bool taskCompleted;

void setup()
{

  Serial.begin(115200);

  WiFi.begin(WIFI_SSID, WIFI_PASSWORD);

  Serial.print("Connecting to Wi-Fi");
  while (WiFi.status() != WL_CONNECTED)
  {
    Serial.print(".");
    delay(300);
  }
  Serial.println();
  Serial.print("Connected with IP: ");
  Serial.println(WiFi.localIP());
  Serial.println();
  Serial.printf("Firebase Client v%s\n\n", FIREBASE_CLIENT_VERSION);

  config.api_key = API_KEY;
  auth.user.email = USER_EMAIL;
  auth.user.password = USER_PASSWORD;
  config.database_url = DATABASE_URL;

  config.token_status_callback = tokenStatusCallback;

  Firebase.begin(&config, &auth);

  Firebase.reconnectWiFi(true);

  fbdo.setBSSLBufferSize(12000, 512);
}

// The Firebase download callback function
void rtdbDownloadCallback(RTDB_DownloadStatusInfo info)
{
  if (info.status == fb_esp_rtdb_download_status_init)
  {
    Serial.printf("Downloading file %s (%d) to %s\n", info.remotePath.c_str(), info.size, info.localFileName.c_str());
  }
  else if (info.status == fb_esp_rtdb_download_status_download)
  {
    Serial.printf("Downloaded %d%s\n", (int)info.progress, "%");
  }
  else if (info.status == fb_esp_rtdb_download_status_complete)
  {
    Serial.println("Backup completed\n");
  }
  else if (info.status == fb_esp_rtdb_download_status_error)
  {
    Serial.printf("Download failed, %s\n", info.errorMsg.c_str());
  }
}

// The Firebase upload callback function
void rtdbUploadCallback(RTDB_UploadStatusInfo info)
{
  if (info.status == fb_esp_rtdb_upload_status_init)
  {
    Serial.printf("Uploading file %s (%d) to %s\n", info.localFileName.c_str(), info.size, info.remotePath.c_str());
  }
  else if (info.status == fb_esp_rtdb_upload_status_upload)
  {
    Serial.printf("Uploaded %d%s\n", (int)info.progress, "%");
  }
  else if (info.status == fb_esp_rtdb_upload_status_complete)
  {
    Serial.println("Restore completed\n");
  }
  else if (info.status == fb_esp_rtdb_upload_status_error)
  {
    Serial.printf("Upload failed, %s\n", info.errorMsg.c_str());
  }
}

void loop()
{

  if (Firebase.ready() && !taskCompleted)
  {
    taskCompleted = true;

    FirebaseJson data;

    // Prepare node data to test
    Serial.println("\nPrepare data... ");

    // Warning memory allocation error can be occurred
    for (int i = 0; i < 8; i++)
    {
      data.set("/data/group" + String(i) + "/a", "________________________________________________________");
      data.set("/data/group" + String(i) + "/b", "________________________________________________________");
      data.set("/data/group" + String(i) + "/b", "________________________________________________________");
      data.set("/data/group" + String(i) + "/c", "________________________________________________________");
      data.set("/data/group" + String(i) + "/d", "________________________________________________________");
      data.set("/data/group" + String(i) + "/e", "________________________________________________________");
      data.set("/data/group" + String(i) + "/f", "________________________________________________________");
      data.set("/data/group" + String(i) + "/g", "________________________________________________________");
      data.set("/data/group" + String(i) + "/h", "________________________________________________________");
      data.set("/data/group" + String(i) + "/i", "________________________________________________________");
      data.set("/data/group" + String(i) + "/j", "________________________________________________________");
      data.set("/data/group" + String(i) + "/k", "________________________________________________________");
      data.set("/data/group" + String(i) + "/l", "________________________________________________________");
      data.set("/data/group" + String(i) + "/m", "________________________________________________________");
    }

    Serial.println("\nSet data... ");
    if (!Firebase.setJSON(fbdo, "/test_backup", data))
      Serial.println(fbdo.errorReason());

    Serial.println("\nBackup test...\n");
    if (Firebase.backup(fbdo, StorageType::FLASH, "/test_backup", "/test.json", rtdbDownloadCallback))
    {
      Serial.printf("backup file, %s\n", fbdo.getBackupFilename().c_str());
      Serial.printf("file size, %d\n", fbdo.getBackupFileSize());
    }
    else
      Serial.println(fbdo.fileTransferError().c_str());

    Serial.println("\nRestore test... \n");

    if (!Firebase.restore(fbdo, StorageType::FLASH, "/test_backup", "/test.json", rtdbUploadCallback))
      Serial.println(fbdo.fileTransferError().c_str());
  }
}

[mobizt]

Result of the test code.

Connecting to Wi-Fi...........
Connected with IP: 192.168.137.54

Firebase Client v4.3.14

Token info: type = id token (GITKit token), status = on request
Token info: type = id token (GITKit token), status = ready

Prepare data...

Set data...

Backup test...

Downloaded 15%
Downloaded 30%
Downloaded 46%
Downloaded 61%
Downloaded 77%
Downloaded 92%
Downloaded 100%
Backup completed

backup file, /test.json
file size, 6650

Restore test...

Uploading file /test.json (6650) to /test_backup
Uploaded 7%
Uploaded 15%
Uploaded 23%
Uploaded 30%
Uploaded 38%
Uploaded 46%
Uploaded 53%
Uploaded 61%
Uploaded 69%
Uploaded 76%
Uploaded 84%
Uploaded 92%
Uploaded 99%
Uploaded 100%
Restore completed

[b1n0-kun]

Hi Mobitz. I tried your code and yes, it can download up to 12kB content ( with your sample, up to group 13). I will try again get it to work with my code. Another irrelevant issue, the stream pause + stream.clear() you showed me in previous topic to gain memory, I couldn't get it resumed, for fbdo pause and resume, it worked.

[mobizt]

I case of stream, the pointers to callback functions were removed after using stream.clear();.

To resume the callback, you can set the callback function again with following.

fbdo.pauseFirebase(false);
stream.pauseFirebase(false);
Firebase.setStreamCallback(stream, streamCallback, streamTimeoutCallback);

FYI When ESP8266 core update is available (newer than current release v3.1.2), stream time out checking will be better (nearly real-time server connection tracking) as it relies on TCP Keepalive by calling stream.keepAlive as in the new stream examples.

[b1n0-kun]

Sorry for a late feedback, I successfully set the backup with raising the BearSSL Buffersize. My NodeMCU get overflowed when I tried to set those buffer size to12k-512b for fbdo and 5k-1024 for stream, then I lower the fbdo to 10k, seems OK now. Can these settings can be change schedulely for backup node ? (raise before and lower after task ?)

Thanks for the suggestion about stream resuming, will test it asap, my daily work get busier and can only touch the esp less than 1 hour/day

[mobizt]

Yes, you can change BearSSL buffer size to any value before opening the session (starting new server connection). The new BearSSL buffer memory allocation with new size will take effect when current or last session was closed and re-open.

Note that, the session will keep alive (reuse) in this library to reduce the SSL handshake waiting time during connecting to server.

To close current session, use fbdo.clear().