custom operation output can exceed Cumulocity's max allowed payload size

[reubenmiller]

Describe the bug

Custom Cumulocity operations which produce a large result (output) and makes heavy usage of double quotes (") will result in publishing an MQTT message to Cumulocity IoT where the payload size exceeds the 16K limit.

This results in the operation being stuck in EXECUTING as the message is rejected by Cumulocity IoT and ultimately causes MQTT connection issues.

After reviewing the code, whilst the output of the custom operation is being truncated to 16000 bytes, the truncation is occurring before the CSV serialization. During the CSV serialization, the payload size can increase a single double quote (") is expanded to two double quotes ("").

The following shows the sequence in which the custom operation output is processed:

1. Sanitize output by removing unsupported control characters
2. Truncate output to the first 16000 bytes
3. Convert to CSV <-- Problem step! Size increases from 16000 to 16000 + total number of double quotes

To Reproduce

1. Install the c8y-command-plugin community plugin

2. Send a shell operation where more than ~600 double quotes are used (so that the total payload is greater than size limit (at least ~17K to be safe))

   yes 'hello"' | head -n 100000

   The above command will result in trying to publish an MQTT message to c8y/s/us where the payload is about 18303 bytes, which is above the Cumulocity IoT default MQTT payload size limit of ~16K.

3. Wait for the operation to be set to SUCCESSFUL. The bug is here, as the operation will be stuck in the EXECUTING state.

Fixing after the test

Since the test is destructive, to restore the system to normal function you have to do the following:

tedge disconnect c8y
systemctl stop mosquitto
rm /var/lib/mosquitto/mosquitto.db
tedge reconnect c8y

Then cancel the operation in Cumulocity IoT by using go-c8y-cli, where {mydevice} is the name or id of the device:

c8y operations list --device "{mydevice}" --fragmentType c8y_Command --status EXECUTING \
| c8y operations cancel

Expected behavior

The total payload of an operation should be truncated after the CSV serialization has occurred.

Screenshots

Environment (please complete the following information):

• OS [incl. version]: any
• Hardware [incl. revision]: any
• System-Architecture [e.g. result of "uname -a"]: any
• thin-edge.io version [e.g. 0.1.0]: 1.3.1

Additional context

The following shows the c8y/s/us messages being published in the above steps. Note the size of the total payload of the 503 message:

C8Y_COMMAND_PAYLOAD.txt

[Bravo555]

Reproduced the issue, but a command to cancel operation in c8y fails with error:

2024-10-09T16:51:03.286+0200    ERROR   commandError: unknown flag: --fragment

c8y version returns 2.43.2 though, which should be current.

[reubenmiller]

Reproduced the issue, but a command to cancel operation in c8y fails with error:

2024-10-09T16:51:03.286+0200  ERROR   commandError: unknown flag: --fragment

c8y version returns 2.43.2 though, which should be current.

My fault, it should be --fragmentType not --fragment. I've updated the instructions now.

[gligorisaev]

Bug is not reproducable anymore