RTDB.setQueryIndex() fails with "security rules is not a valid JSON"

Hi, big thanks for all the hard work on this library.

I was going through the examples when RTDB/DataRetaining yielded this.

Describe the bug

Connecting to Wi-Fi..................
Connected with IP: 192.168.1.137

Firebase Client v4.4.8

Token info: type = id token (GITKit token), status = on request
Token info: type = id token (GITKit token), status = ready
Set query index in database rules... security rules is not a valid JSON
Push data... ok
Delete history data older than 10 minutes... Index not defined, add \".indexOn\": \"ts\", for path \"/test/log\", to the rules
Push data... ok

etc.

To Reproduce examples/RTDB/DataRetaining/DataRetaining.ino, with appropriate WiFi and Firebase secrets.

Expected behavior Database rules modified succesfully.

IDE and its version: Arduino CLI 0.32.2 via VS Code Arduino extension

ESP8266 Arduino Core SDK version 3.1.2

Additional context Board: WEMOS D1 mini (clone?) based on ESP-12F

It's not issue. It is library intended error to warn user that his security rule can't be processed.

As the error shown you should make your security rules valid JSON before using this function.

Copy your security rules and paste into this tool and check what causes your rules invalid. https://jsonformatter.curiousconcept.com/

It seems as if you assume my rules are actually invalid JSON. I don't think that holds.

The initial report used these rules

{
  "rules": {
    ".read": "auth != null", 
    ".write": "auth != null"
  }
}

Even if I change them to

{
  "rules": {
    "test": {
      "log": {
        ".indexOn": "ts",
        ".read": "auth != null",
        ".write": "auth != null",
      }
    }
  }
}

Said example still gives me the error:

Connecting to Wi-Fi........
Connected with IP: 192.168.1.137

Firebase Client v4.4.8

Token info: type = id token (GITKit token), status = on request
Token info: type = id token (GITKit token), status = ready
Set query index in database rules... security rules is not a valid JSON
Push data... ok
Delete history data older than 10 minutes... ok
Push data... ok
Push data... ok
Delete history data older than 10 minutes... ok
Push data... ok

etc.

I confirm that everything is ok.

Result of DataRetaining example

Connecting to Wi-Fi.....
Connected with IP: 192.168.71.141

Firebase Client v4.4.8

Token info: type = id token (GITKit token), status = on request
Token info: type = id token (GITKit token), status = ready
Set query index in database rules... ok
Push data... ok
Delete history data older than 10 minutes... ok
Push data... ok
Push data... ok
Delete history data older than 10 minutes... ok
Push data... ok
Push data... ok
Delete history data older than 10 minutes... ok
Push data... ok
Push data... ok

Result of DataFilter example

Connecting to Wi-Fi.....
Connected with IP: 192.168.71.141

Firebase Client v4.4.8

Token info: type = id token (GITKit token), status = on request
Token info: type = id token (GITKit token), status = ready
Push json... ok
Push json... ok
Push json... ok
Push json... ok
Push json... ok
Push json... ok
Push json... ok
Push json... ok
Push json... ok
Push json... 
Push json... ok
Push json... ok
Push json... ok
Push json... 
Push json... ok
Push json... ok
Push json... ok
Push json... 
Push json... ok
Push json... ok
Push json... ok
Push json... 
Push json... ok
Push json... ok
Push json... ok
Push json... 
Push json... ok
Push json... ok
Push json... ok
Push json... 

Get json... ok
Pretty printed JSON data:
{
        "-NeOKGyEM5gAZvNONNaG": {
                "Data1": 20,
                "Data2": 119
        },
        "-NeOKGzIEeIRUOta6USe": {
                "Data1": 21,
                "Data2": 120
        },
        "-NeOKaWwgfi0mHEBO41J": {
                "Data1": 19,
                "Data2": 118
        },
        "-NeOKaY3hEWyaISGMmop": {
                "Data1": 20,
                "Data2": 119
        },
        "-NeOKaZHwzX88lAqFJ8H": {
                "Data1": 21,
                "Data2": 120
        },
        "-NeOM41OatWnYwXGLIP-": {
                "Data1": 19,
                "Data2": 118
        },
        "-NeOM42NNlmOEXJZ6BbV": {
                "Data1": 20,
                "Data2": 119
        },
        "-NeOM43oGLOJvSDM-FTf": {
                "Data1": 21,
                "Data2": 120
        }
}
Iterate JSON data:

0, Type: object, Name: -NeOKGyEM5gAZvNONNaG, Value: {"Data1":20,"Data2":119}
1, Type: object, Name: Data1, Value: 20
2, Type: object, Name: Data2, Value: 119
3, Type: object, Name: -NeOKGzIEeIRUOta6USe, Value: {"Data1":21,"Data2":120}
4, Type: object, Name: Data1, Value: 21
5, Type: object, Name: Data2, Value: 120
6, Type: object, Name: -NeOKaWwgfi0mHEBO41J, Value: {"Data1":19,"Data2":118}
7, Type: object, Name: Data1, Value: 19
8, Type: object, Name: Data2, Value: 118
9, Type: object, Name: -NeOKaY3hEWyaISGMmop, Value: {"Data1":20,"Data2":119}
10, Type: object, Name: Data1, Value: 20
11, Type: object, Name: Data2, Value: 119
12, Type: object, Name: -NeOKaZHwzX88lAqFJ8H, Value: {"Data1":21,"Data2":120}
13, Type: object, Name: Data1, Value: 21
14, Type: object, Name: Data2, Value: 120
15, Type: object, Name: -NeOM41OatWnYwXGLIP-, Value: {"Data1":19,"Data2":118}
16, Type: object, Name: Data1, Value: 19
17, Type: object, Name: Data2, Value: 118
18, Type: object, Name: -NeOM42NNlmOEXJZ6BbV, Value: {"Data1":20,"Data2":119}
19, Type: object, Name: Data1, Value: 20
20, Type: object, Name: Data2, Value: 119
21, Type: object, Name: -NeOM43oGLOJvSDM-FTf, Value: {"Data1":21,"Data2":120}
22, Type: object, Name: Data1, Value: 21
23, Type: object, Name: Data2, Value: 120

Database security rules.

{
    "rules": {
        ".read": "auth != null",
        ".write": "auth != null",
        "test": {
            "log": {
                ".indexOn": "ts"
            },
            "push": {
                ".indexOn": "Data2"
            }
        }
    }
}

Oh, I see... trailing commas are not JSON in the RFC 8259 sense. Not very helpful as Firebase console defaults to using them :facepalm:

README, under The authenication credentials and prerequisites:

The Firebase RTDB security rules are JSON-based rules which it should valid to used with this library RTDB functions that involved the security rules modification and reading, otherwise the rules wont be changed or read by these functions.

Clarifying this might benefit future users stumbling on this. How about something like this instead?

The Firebase RTDB security rules are JSON-based rules modifiable through the Firebase console and the REST API. While the Firebase console defaults to a less strict JSON syntax, In order to read or modify the rules with this library, rules already present need to adhere to RFC 8259, i.e. no trailing commas allowed.

Your database rules are invalid in FirebaseJson implementation and it is not allowing incomplete, extra character other than key-value pair that should be and well-formed data.

Security rules are similar to JSON but not JSON especially in Firestore then we can't assume it to be JSON but the rules for RTDB are simpler and are very similar to JSON and if user need to use the function to modify rules, he should make his rules valid JSON.

Alright, In hindsight, I should have created a discussions thread instead to figure out the conflicting comma.

FWIW, I still think clarifying the README would be beneficial, especially as the screenshots^{1, 2} depict the unfortunate decision on Firebase's part to default to nonstandard JSON. Although I now understand the intended meaning, the sentence I quoted above does not parse very well according to my understanding of the English language.

Maybe all RTDB examples operating on the security rules could also have a comment mention that the default rules provided by Firebase will result in an error.

If you think these points are worth acting on, I can definitely donate a bit of my time to make a PR.

Thanks for your quick responses and my apologies for the hassle.

I appreciate your time for contribution.

The new version 5 is coming with breaking changes which affected to the feature you used.

The new asynchronous operation, code size, program footprint and variety of functions that provided by the library development in the past are the keys for code refactoring and cutting in version 5.

Here is the information of all changes that break the old version compatibility and will be stated on Readme.

Breaking Changes from version 4

To keep support only necessary options and features that provided by Firebase which can be reduced the program footprint, there are some functions removed from this library in version 5.

Note that, the following changes applied for Firebase-ESP-Client, Firebase-ESP8266 and Firebase-ESP32 libraries.

The functions removal.

The MultiPath Streaming in RTDB.

beginMultiPathStream, setMultiPathStreamCallback, and removeMultiPathStreamCallback
The Streaming in RTDB.

setStreamCallback and removeStreamCallback
The backup and restore in RTDB.

backup and restore
The error queue and its data type in RTDB.

setMaxErrorQueue, saveErrorQueue, deleteStorageFile, restoreErrorQueue, errorQueueCount, isErrorQueueFull, processErrorQueue, getErrorQueueID, isErrorQueueExisted, beginAutoRunErrorQueue, endAutoRunErrorQueue, and clearErrorQueue
The security rules download, upload and set the read/write rules in RTDB.

getRules, setRules and setReadWriteRules.
The query index set/remove in RTDB.

setQueryIndex and removeQueryIndex
The anync functions in RTDB.

setPriorityAsync, pushAsync, pushIntAsync, pushFloatAsync, pushDoubleAsync, pushBoolAsync, pushStringAsync, pushJSONAsync, pushArrayAsync, pushBlobAsync, pushFileAsync, pushTimestampAsync, setAsync, setIntAsync, setFloatAsync, setDoubleAsync, setDoubleAsync, setBoolAsync, setStringAsync, setJSONAsync, setArrayAsyn, setBlobAsync, setFileAsync, setTimestampAsync, updateNodeAsync, and updateNodeSilentAsync
The node deletion in RTDB.

deleteNodesByTimestamp
FireSense add on in RTDB.
Old API in FCM for ESP32 and ESP8266 version library.

fcm.begin, fcm.addDeviceToken, fcm.removeDeviceToken, fcm.clearDeviceToken, fcm.setNotifyMessage, fcm.addCustomNotifyMessage, fcm.setDataMessage, fcm.clearDataMessage, fcm.setPriority, fcm.setCollapseKey, fcm.setTimeToLive, fcm.setTopic, fcm.getSendResult, sendMessage, broadcastMessage and sendTopic
The async function in Cloud Firestore.

runDeployTaskscommitDocumentAsync
Old API in RTDB for ESP32 and ESP8266 version library.

setReadTimeout, setwriteSizeLimit, pathExisted, pathExist, getETag, getShallowData, enableClassicRequest, setPriority, getPriority, push, pushInt, pushFloat, pushDouble, pushBool, pushString, pushJSON, pushArray, pushBlob, pushFile, pushTimestamp, setInt, setFloat, setDouble, setBool, setString, setJSON, setArray, setBlob, setFile, setTimestamp, updateNode, updateNodeSilent, get, getInt, getFloat, getDouble, getBool, getString, getJSON, getArray, getBlob, getFile, downloadOTA, deleteNode, beginStream, readStream, runStream, and setMaxRetry

Note that these functions should be called from RTDB member class of Firebase class and use the pointer to FirebaseData, FirebaseJson and FirebaseJsonArray in the functions.
The legacy functions in FirebaseData

intData, floatData, doubleData, boolData, stringData, jsonString, jsonObject, jsonObjectPtr, jsonArray, jsonArrayPtr, blobData, and fileStream
The external Client functions.

setExternalClient and setExternalClientCallbacks

The enums and struct removal

This removal is for proper naming.

The fb_esp_rtdb_data_type enums in RTDB

fb_esp_rtdb_data_type_null, fb_esp_rtdb_data_type_integer, fb_esp_rtdb_data_type_float, fb_esp_rtdb_data_type_double, fb_esp_rtdb_data_type_boolean, fb_esp_rtdb_data_type_string, fb_esp_rtdb_data_type_json, fb_esp_rtdb_data_type_array, fb_esp_rtdb_data_type_blob, and fb_esp_rtdb_data_type_file
The fb_esp_rtdb_upload_status enums in RTDB

fb_esp_rtdb_upload_status_error, fb_esp_rtdb_upload_status_unknown, fb_esp_rtdb_upload_status_init fb_esp_rtdb_upload_status_upload, and fb_esp_rtdb_upload_status_complete
The fb_esp_rtdb_download_status enums in RTDB

fb_esp_rtdb_download_status_error, fb_esp_rtdb_download_status_unknown, fb_esp_rtdb_download_status_init fb_esp_rtdb_download_status_download,and fb_esp_rtdb_download_status_complete
The fb_esp_cfs_upload_status enums in Cloud Firestore

fb_esp_cfs_upload_status_error, fb_esp_cfs_upload_status_unknown, fb_esp_cfs_upload_status_init fb_esp_cfs_upload_status_upload,fb_esp_cfs_upload_status_complete, and fb_esp_cfs_upload_status_process_response
The fb_esp_gcs_upload_status enums in Google Cloud Storage

fb_esp_gcs_upload_status_error, fb_esp_gcs_upload_status_unknown, fb_esp_gcs_upload_status_init fb_esp_gcs_upload_status_upload, and fb_esp_gcs_upload_status_complete
The fb_esp_gcs_download_status enums in Google Cloud Storage

fb_esp_gcs_download_status_error, fb_esp_gcs_download_status_unknown, fb_esp_gcs_download_status_init fb_esp_gcs_download_status_download, and fb_esp_gcs_download_status_complete
The fb_esp_fcs_upload_status enums in Firebase Storage

fb_esp_fcs_upload_status_error, fb_esp_fcs_upload_status_unknown, fb_esp_fcs_upload_status_init fb_esp_fcs_upload_status_upload, and fb_esp_fcs_upload_status_complete
The fb_esp_fcs_download_status enums in Firebase Storage

fb_esp_fcs_download_status_error, fb_esp_fcs_download_status_unknown, fb_esp_fcs_download_status_init fb_esp_fcs_download_status_download, and fb_esp_fcs_download_status_complete
The fb_esp_functions_operation_status enums in Cloud Functions for Firebase

fb_esp_functions_operation_status_unknown, fb_esp_functions_operation_status_generate_upload_url, fb_esp_functions_operation_status_upload_source_file_in_progress fb_esp_functions_operation_status_deploy_in_progress, fb_esp_functions_operation_status_set_iam_policy_in_progress, fb_esp_functions_operation_status_delete_in_progress, fb_esp_functions_operation_status_finished and fb_esp_functions_operation_status_error

Note that, you can replace the enums fb_esp_xxx with firebase_xxx.

The StorageType struct for ESP32 and ESP8266 version library.

StorageType::UNDEFINED, StorageType::FLASH and StorageType::SD

Note that, mem_storage_type_undefined, mem_storage_type_flash, and mem_storage_type_sd was used instead.

The platform and its features removal.

Because of no benefit we gain from using FreeRTOS in Raspberry Pi Pico platform in this library and limited resources and C++ functions supported in AVR platform, the following are changes.

FreeRTOS removal in Raspberry Pi Pico in Google Cloud Storage upload task.
FreeRTOS removal in Raspberry Pi Pico in Functions for Firebase task.

createFunction and runDeployTasks
The std::vecor was used to represent the dynamic array and it breaks the AVR platform supports.

To bring back the removed functions.

You should copy the relevant function you used in the source code of version 4 (mostly in FB_RTDB.h and FB_RTDB.cpp) to your sketch.

The removed functions implemented using the library public functions and some private helper functions.

mobizt / Firebase-ESP-Client