Sending images to the Thingsboard Edge DB- problem

[jose-franco-PT]

Description Good Morning,

I have a raspberry Pi device that is running thingsboard edge successfully.

Right now, I am doing an experiment which would be to send images to a device on Thingsboard Edge. Therefore I created a python script on the Raspberry Pi that takes an image (640X450), transforming it into a base64 string and then sends that string in the mqtt payload:

Unfortunately I noticed that this string was not being received in the Thingsboard telemetry data. Checking the error logs I see the following error:

2023-03-29 12:22:09,146 [nioEventLoopGroup-4-8] ERROR o.t.s.t.mqtt.MqttTransportHandler - [ae237b7c-7a7e-4508-b9d3-cce29b1bd747] Message decoding failed: too large message: 146427 bytes

I hereby come to ask if it will be possible to increase the size of this mqtt buffer of Thingsboard Edge? Otherwise what is the limit?

Thank you for your attention

Environment

• OS: Windows 10 64 bit
• ThingsBoard: 3.4.4 (Docker Version)
• Browser: name and version

[volodymyr-babak]

@jose-franco-PT

you are interested in NETTY_MAX_PAYLOAD_SIZE. By default it's 65536. https://thingsboard.io/docs/edge/user-guide/install/config/#local-mqtt-transport-parameters

Please add next line into tb-edge.conf and restart edge service: export NETTY_MAX_PAYLOAD_SIZE=2000000

The same configuration is applicable for ThingsBoard server, if required.

[volodymyr-babak]

@jose-franco-PT

but please make sure that you are not sending this telemetry to the cloud. I believe that size of this payload is going to exceed 4 MB. And this will hit limits of maximum size of gRPC packet.

[jose-franco-PT]

Good afternoon, I added the statement export NETTY_MAX_PAYLOAD_SIZE=2000000 to the tb-edge.conf file. However, unfortunately the message is still unobserved in "last Telemetry" on the devices page in Thingsboard Edge.

On the other hand, when I run the statement cat /var/log/tb-edge/tb-edge.log | grep ERROR from the command line on the Raspberry Pi, I see the following errors:

I wonder if I am missing some procedure.Thank you for your attention.

[volodymyr-babak]

@jose-franco-PT

I believe that size of the data, that you are trying to store into PostgreSQL is too big. Please check logs of PostgreSQL service - I believe you'll find errors there regarding message size.

[jose-franco-PT]

I've been checking these two files:

However the information presented does not allow me to draw any conclusions unfortunately:

Even through the journalctl command no relevant information is displayed either:

-Is there another place where I can check the logs?

[volodymyr-babak]

@jose-franco-PT

that is the correct PostgreSQL log file. Could you please attach to the ticket full log of tb-edge from yesterday? The one that contains [Attributes] Failed to save 1 entities. According to the implementation we should see stack trace of the exception in the log.

[jose-franco-PT]

Good morning, tb-edge.2023-03-29.0.log

I apologize for only replying now. I think it is the file that is attached. Thank you for your attention

[volodymyr-babak]

@jose-franco-PT

Thanks for the logs, but these errors are related to the fact that TB Edge service was stopped, and not because of impossibility to save something into the database. Did you test the same payload against ThingsBoard Community Edition - are you able to store these huge timeseries successfully in the case of CE?

[jose-franco-PT]

Good afternoon,

I just found out what the root of the problem is. Essentially I printed out the size of the string, giving 146391 characters. Next I generated a string with the same size but with the characters a-z( as well as uppercase) and the digits 0-9.

Then with this string I was already able to send to the Thingsboard Edge.

Then I discovered that the characters "/" and "=" are not accepted. Basically when I do the conversion to base64 string those two characters appeared. If I replace them with 2 others, the string is sent successfully.

Thanks.

[volodymyr-babak]

@jose-franco-PT

thanks for the update. In this case I'm closing this ticket, but please feel free to re-open it in case new issues or questions.