HELP: File size during PUT and GET operations

[usrdes]

Hi Mobizt, In the File.ino example, you have the comments indicating file size > 10K leads to unexpected results.

However, I tested with 16K, and the file write (PUT) appears to be OK. But GET operation truncates at ~11K, as you mention.

I am wondering if it is due to SD being slow? Does your library de-couple the firebase read and SD write operations?

Thank you

  /*/////////////////////////////////////////////////////
  Sending large file ( > 10k) leads to unexpected result
  /////////////////////////////////////////////////////*/

  //Write demo data to file (8192 bytes)
  file = SD.open("/source.txt", FILE_WRITE);
  for (int i = 0; i < 8192; i++)
    file.write(0xf);

  file.close();

[mobizt]

It's not depend on storage, the problem is we get never ending stream.

The SSL client (mbedTLS) rx buffer still return more byte of data (value other than -1 which indicated fail to read data from buffer) which greater than the value of Content length header.

When store large string (base64) which is bigger than 10k, we don't know actually the size of data stored on database, or the way to get it from textbox in console to validate our data from store operation, and we get never ending stream response when read that data.

[usrdes]

OK, thank you for the quick response Regards

[usrdes]

Sorry, closed too soon.. One more question: What if we know ahead of time the file size, or set the size to a value in one of the "key - value" pairs as part of the db. Yes this would mean two "GET" operations but may be a work around?

I am attempting to send / receive ~24KB data. So if the limit is 8KB, I would need to break up the file and send three chunks of 8KB each, and then when I get it back, re-assemble the three chunks ...

[mobizt]

You can mix the file size, file name and file part number as part of the node path i.e. /Foo/image_1_208k for file named image, part 1 of 20, size 8 kb. Which you can split the node path or name with to git info from it.

[usrdes]

How about if the file is audio. Does this library support audio streaming? I noticed that in your streaming example, there is no firebaseData.dataType() == "file", only "blob". But if the file is audio, I can stream the file to an ESP32 audio player directly without any worries of file size right?

[mobizt]

No. No matter your data is raw PCM (wave) or MPEG encoded or any byte format, you will need to convert it to String which support by Firebase. The problem when your data that has byte value equal to ASCII control code, it affected by server to decode that byte into valid data. This is reason why that base64 string is safe to transfer to all http server that accept text only because it contains a-z, A-Z,0-9 and padding char =. Base64 is commonly used in email attachment which all byte of your attachment will be convert to base64 string

[usrdes]

Hi Mobizt, Yes I understand that the base64 should be used -- OK. My main question is : If the file is large, can your library support streaming the file from Firebase?

Maybe I am not understanding "streaming" properly: "streaming" is not actually sending a stream, but more of a "listener" for a change of value on an object stored in the firebase realtime data base. Is this correct? If so there can be no real stream of data being sent from firebase. The code on ESP32 will simply need to read the file out of the realtime db. But due to file size limitations, it needs to be less than 8KB (with base 64, it might go up to 10KB max).

[mobizt]

Actually yes for http stream, but there are some limitation to handle the payload due to Firebase REST API returns all changed data as payload.

The stream payload can be any primitive data type included json object. The string, number, boolean payloads are easy to handle.

For json object payload, it causes the library to parse all key/value and convert base64 string to binary, which is not possible becasue it needs atlease two times memory to buffer and make complicated implementation of my library API and may not compattible to the API concept.

Summary, if your base64 data changed, you will get base64 string or json object contains base64 string instead of binary data from stream payload.

[mobizt]

I will declare the http stream you asked above, client (library) not need to poll or send any request to server periodically to get data, the data will push from server directly as long as your device is connected.

The library itself also looking for keep-alive payload from server to make sure that server sill be accessible or connected to device during stream, that's it.

The library not need to send any request to read the data from server unless the keep alive payload is absent more than 30 sec then re-establish the connection again and waiting for server payload.

Firebase send keep-alive payload periodically but the interval may be varied due to other factors.

The data size of stream payload is not limit, but from implementation of large string (base64) more than 10k, we get contaminated data and makes the never ending payload.

[usrdes]

"I will declare the http stream you asked above," -- You mean you will update the library?

But even if the streaming works as the word (streaming) implies: i.e. Firebase streams data to ESP32, it will still have the "10K" limitation. Right? And it looks like Google firebase needs to "remove" this limitation of contaminated data for greater than 10KB.

So, from a performance perspective: If the streaming works by Firebase streaming the data to ESP32, will this be "faster" than Firebase.getFile(firebaseData.... ) ?

Once again thank you for your kind responses

[mobizt]

No document for max data size store in each node then don't judge 10k as limitation as we cannot validate the data store on db (it contaminated or not) and we can store more than 10k data without reject as we get http return code 200. In Firestore, 1 MB of each document is clearly stated.

I don't know "faster" you mean, but stream data is realtime (as MQTT protocol) because server push data by itself and reduce the client tasks for polling or request to get fresh data.

If you polling server data by periodically read the data every 3 seconds, it's a delay to check update atleast 3 sec compare to realtime streaming data.

Firebase RTDB and Firestore are no sql database not data storage service, please don't try to put everything to them and judge that they din't work or limited.