JSON parser error with colon symbol

[UlisesGascon]

Expected Behavior

This manifest should work because is a valid JSON and a valid release content, but if you include : in the name property it won't work as the du won't parse it properly.

{
    "name": "Corretto machine: initial manifest",
}

Current Behavior

DU Json parser should support : in the properties.

Steps to Reproduce

Provide a detailed set of steps to reproduce the bug.

1. Follow this steps but include : in the name property in this file: corretto-1.0.2-manifest.json

Device Information

• Host OS: Ubuntu 18.04
• Architecture: amd64
• Provisioning Method: ? *DU Agent Version (run 'sudo -u adu /usr/bin/AducIotAgent -v'): 1.0.2

Logs

Additional Information

Please provide any additional information that may be helpful in understanding the issue.

[andrewbrownmsft]

Hi. Where is the failure occurring? Is it when importing the update into the DU service, or on the device?

[UlisesGascon]

Hi @andrewbrownmsft :)

On the device, the importation worked fine.

[andrewbrownmsft]

Got it, that's what I thought but wanted to be sure. In that case, @eshashah-msft, can you take a look here?

[jw-msft]

@UlisesGascon Is this the top-level "name" property or the "name" properties in the packages array elements? It's unclear if it's the top-level name property or those in the packages array and if it's the property name or the property value.

Could the repro .json file be provided?

The "Failed to parse specified APT file" trace occurs when the Parson library json_parse_file() call returns NULL here: https://github.com/Azure/iot-hub-device-update/blob/b71516d3e85f09db48c06980448e3a07d90d05e4/src/extensions/step_handlers/apt_handler/src/apt_parser.cpp#L128

I created the following test app that just calls json_parse_file() and returns 0 on success or 1 on failure:

#include <parson.h>
int main(int argc, char**argv) {
    return NULL == json_parse_file(argv[1]) ? 1 : 0;
}

and built it with: gcc -g main.c -lparson

After a wget https://raw.githubusercontent.com/UlisesGascon/poc-azure-iot-du-custom-package/main/corretto-1.0.2-manifest.json, I did:

$ ./a.out corretto-1.0.2-manifest.json
$ echo $?
0

I tried making the following changes, but they also all returned 0 (success):

• "name" to "na:me"
• "Corretto machine initial manifest" to "Corr:etto machine initial manifest"
• "name" to "na:me" in packages array
• "demoapi" to "demo:api"

[UlisesGascon]

Thanks for all the work and the research @jw-msft! Can you try with this one?

[jw-msft]

Downloaded that file and test app succeeded (parson parses it successfully) -- I see it has the colon char in the value of top-level name property and test app succeeded(0), which should be fine.

Next, I did an E2E repro with released Ubuntu 18.04 DU Agent, but was also not able to repro.

The only thing I can think of that might lead to the issue you're seeing is if you had an old/buggy version of built libparson.a static lib installed on the system in the lib path and then built DU agent from sources with that, or maybe something's introducing some weird non-printable character in the APT manifest.json right before you're uploading it to azure portal blob storage.

E2E Repro Attempt

Here is how I went about the repro attempt:

• Created a new Ubuntu 18.04.6 LTS VM using https://cdimage.ubuntu.com/releases/18.04/release/ubuntu-18.04.6-server-amd64.iso linked from https://cdimage.ubuntu.com/releases/18.04/release/

• Installed DeliveryOptimization v1.0.0 agent dependency from .deb pkg on packages.microsoft.com:

wget https://packages.microsoft.com/ubuntu/18.04/multiarch/prod/pool/main/d/deliveryoptimization-agent/libdeliveryoptimization_1.0.0_amd64.deb
wget https://packages.microsoft.com/ubuntu/18.04/multiarch/prod/pool/main/libd/libdeliveryoptimization/libdeliveryoptimization_1.0.0_amd64.deb
sudo apt-get install ./deliveryoptimization-agent/libdeliveryoptimization_1.0.0_amd64.deb
sudo apt-get install ./libdeliveryoptimization_1.0.0_amd64.deb

• Installed released DeviceUpdate v1.0.2 agent from .deb pkg on packages.microsoft.com:

wget https://packages.microsoft.com/ubuntu/18.04/multiarch/prod/pool/main/d/deviceupdate-agent/deviceupdate-agent_1.0.2_ubuntu1804_amd64.deb
sudo apt-get install ./deviceupdate-agent/deviceupdate-agent_1.0.2_ubuntu1804_amd64.deb

• Setup the /etc/adu/du-config.json with Manufacturer "Contoso" and Model "Toaster"
• Created and uploaded an import manifest (including the apt manifest json that I renamed to value_colon_manifest.json) for APT package update.

Result

As can be seen from du-agent.20230817-192435.log in the zip, the apt json parse issue was not hit and it continued on to download phase of apt package install, invoking apt that attempted to download those packages.

Zip of adu logs from repro attempt and import manifest used:

repro_attempt.zip

Further steps to try

• Build DU agent from sources with debug symbols included

git clone https://github.com/azure/iot-hub-device-update
cd iot-hub-device-update
./scripts/install-deps.sh -a
./scripts/build.sh -c --build-packages
cd out

• sudo apt install the .deb in out

• sudo apt install gdb

• gdb --args /usr/bin/AducIotAgent -l0 -e

• set breakpoint in AptParser::ParseAptContentFromFile and drill down into the parson json_parse_file function

[jw-msft]

Closing due to stale issue and was not able to repro with the latest provided manifest.