How to use serial port used by thingset for data download

[Ananyaaynana]

We want to download to the BMS C1 cell characterization data (that is a few kilo bytes in size) over the same serial port used by thingset.

The cell characterization data consists of 19 arrays of type double. Each array has different number of elements typically exceeding 200.

How will we do this?

[martinjaeger]

You could structure the data into several smaller groups. An you should probably increase the buffer sizes.

Please note that double is currently not supported by the ThingSet node library, but it could be easily extended. Do you really need double or is float sufficient?

Can you give an example of your data struct for better understanding?

[Ananyaaynana]

Please explain as to what you mean by several smaller groups. We thought of using a thingset group to store all the 19 arrays. Would it work if we use 'THINGSET_ADD_ITEM_ARRAY' to store the arrays in the NVM. What value should we increase the buffer size to? We need double data type. The data structure has 19 double arrays. For each array we store the number of elements in the array(as an int) and the array itself.

[martinjaeger]

In principle ThingSet should work with data of any size. The buffer sizes can be adjusted via Kconfig. Did you try it?

[martinjaeger]

I realized that deserialization of arrays was only supported in binary mode so far (because that was what we needed for our projects so far).

I've added the feature to the ThingSet node library, see PR here. The PR also shows the correct syntax in the unit tests.

You can add it to the BMS for testing by adding the following to your west.yml and running west update afterwards:

    - name: thingset-node-c
      remote: thingset
      revision: 4b5809d696033fadd2210388698013646f951911
      path: modules/thingset-node-c
      import: true

The PR will be integrated into the BMS with the next update of the ThingSet SDK.

[Ananyaaynana]

Thank you will integrate this into the firmware and use it.

[Ananyaaynana]

Hello martin. Each array that we want to write to thingset has 200+ elements x(multpily) 18-19 digits per element, the line containing the array will have more than 3600-3800 characters. How do we split this data among multiple lines of input?

We also want to store the arrays into a partition in the flash that I have created. When I use the subset: 'TS_SUBSET_NVM' which partition do the arrays get stored in? How to store the arrays in the partition that I have created so that there is a separation between the cell model data and the other information stored in the NMV about the board such as the protection configurations?

[martinjaeger]

Hello martin. Each array that we want to write to thingset has 200+ elements x(multpily) 18-19 digits per element, the line containing the array will have more than 3600-3800 characters. How do we split this data among multiple lines of input?

You cannot split the line containing the data of one single array. You will have to choose a large enough buffer (e.g. 4000 in your case). And you should probably not do it manually, but instead write a Python script or something like that, which parses your data, strips all newlines, creates the ThingSet request and sends it over the serial (or over CAN).

We also want to store the arrays into a partition in the flash that I have created. When I use the subset: 'TS_SUBSET_NVM' which partition do the arrays get stored in? How to store the arrays in the partition that I have created so that there is a separation between the cell model data and the other information stored in the NMV about the board such as the protection configurations?

The data is stored in the storage_partition (see here). ThingSet will automatically deserialize the data into the flash and restore it (serialize the data from flash) during boot-up. The data is identified based on their numeric ID. This should also work with arrays. Let me know if you have any issues with that (please open a dedicated issue in that case).

[Ananyaaynana]

Hello Martin. I wrote a C program that creates the Thingset request and parses through the 201 elements of the array then writes it to the serial port. I am almost done except for one obstacle.

The array has 201 elements, but the TS terminal stops accepting after 12 elements. I have increased the 'THINGSET_SHARED_TX_BUF_SIZE' to 5000. I could not find a 'THINGSET_SHARED_RX_BUF_SIZE' in the Kconfig file.

When the array has 8 elements I get this error:

In the ~/west-workspace/modules/thingset-node-c/Kconfig.thingset file I increased the 'THINGSET_NUM_JSON_TOKENS' from 50 to 300 thinking that tokens refer to the number of elements in the array. This change did not help.

How do I get past this?

[martinjaeger]

There is also a limit of the maximum size for shell commands (it uses a separate buffer). You can look at the shell code in the ThingSet SDK.

Either increase that buffer or use the ThingSet serial directly (on the UART connector of the BMS C1 or you disable the shell and move it to the USB serial via Devicetree). Or use CAN.

Hope that helps.

[Ananyaaynana]

I changed the THINGSET_SHELL_CMS_BUF_SIZE from 128 to 2048 and flashed the program. A few spaces are because of the format specifier I used for the floating point values to be written to the serial port. I got the above error. The "Too many arguments in the error" error is from the Zephyr shell.

So I changed the size of the SHELL_ARG_MAX from 20 to 255 and the SHELL_CMD_BUFF_SIZE from 256 to 2048. It can accept all 213 elements of the array but after the entire array is written to the serial port I get the below error.

I have checked the syntax by sending an array of 8 elements and can successfully store these elements in the array defined in the thingset. I get this error when I use the same syntax to send 213 elements. The entire array with the closing braces is written to the serial port but it is not displayed on the thingset terminal. Do you have any idea why this is happening?

[martinjaeger]

No, sorry, don't have any idea. Probably the shell is just not made for such huge amounts of data.

[Ananyaaynana]

Changed the CONFIG_THINGSET_NUM_JSON_TOKENS from 50 to 300. This did the trick. Now able to send 200+ floating point elements. Thank you for the help.