Extend c8y-configuration-plugin to support config management in child devices

[albinsuresh]

Is your feature request related to a problem? Please describe.

Support the following config management operations on child devices:

1. Fetch config snapshots from child devices to Cumulocity cloud
2. Push config updates from Cumulocity to child devices

Describe the solution you'd like

Implement the config management solution for child devices described in: https://github.com/thin-edge/thin-edge.io/pull/1236

• The c8y-configuration-plugin to map c8y_ConfigUpload requests (SmartREST 526) to tedge/<child-id>/commands/req/config_snapshot requests and deliver it to the appropriate child device
• Similarly, map c8y_ConfigDownload requests (SmartREST 524) to tedge/<child-id>/commands/req/config_update requests and deliver it to the appropriate child device
• The plugin uses tedge file repository to store the binary files downloaded from Cumulocity before they shared with child devices
• The plugin maps operation status updates that it receives updates from the external device to Cumulocity after mapping them to their SmartREST equivalents.
• The plugin is responsible for deleting any binary files stored in the tedge file repository after an operation completes. That applies to both the files uploaded by the plugin itself (for config_update operation) as well as the file uploaded by the child devices for config_snapshot operation.
• The c8y-configuration-plugin loads the provided config file list for a child-device from a child-devices individual TOML file
• A dummy child device process implemented in any language can be used for testing this feature. Even the mock chil device in https://github.com/thin-edge/thin-edge.io_spikes/pull/4

Out of scope

• Handling of operations that takes too long (failing them with timeouts)

[cstoidner]

I see things we should add to the scope:

• The c8y-configuration-plugin loads the provided config file list for a child-device from a child-devices individual TOML file
• Whenever the c8y-configuration-plugin loads the TOML file, the c8y-configuration-plugin creates the folder /etc/tedge/operations/c8y/<child-id> and creates the supported operations files c8y_UploadConfigFile and c8y_DownloadConfigFile in that folder
• The supported operations API accepts supported operations in a subfolder and delcares corresponding supported operation to the clouds child-device twin, using the subfolder name as child-id
• Whenever the c8y-configuration-plugin loads the TOML file, the c8y-configuration-plugin sends the provided configuration types defined in the child-devices TOML file to the clouds child-device twin.

Also things to add to "out of scope":

• Announcing list of supported configuration files by the child-device itself (to the plugin via MQTT)

[albinsuresh]

I see things we should add to the scope:

• The c8y-configuration-plugin loads the provided config file list for a child-device from a child-devices individual TOML file
• Whenever the c8y-configuration-plugin loads the TOML file, the c8y-configuration-plugin creates the folder /etc/tedge/operations/c8y/<child-id> and creates the supported operations files c8y_UploadConfigFile and c8y_DownloadConfigFile in that folder
• The supported operations API accepts supported operations in a subfolder and delcares corresponding supported operation to the clouds child-device twin, using the subfolder name as child-id
• Whenever the c8y-configuration-plugin loads the TOML file, the c8y-configuration-plugin sends the provided configuration types defined in the child-devices TOML file to the clouds child-device twin.

Also things to add to "out of scope":

• Announcing list of supported configuration files by the child-device itself (to the plugin via MQTT)

The first item has been pulled into the scope of this story. The rest of the items are more about child device provisioning and hence, would be addressed in #1336.

[cmosd]

This test requires a child device.

The child device should have a configuration file like this:

files = [
    { path = '/home/user/Desktop/file_a', type = 'file_a' },
]

The child device also requires a script that communicates with the parent device via MQTT and HTTP. An example script can be found in thin-edge.io_spikes.

Tests should cover:

child device bootstrapping

On the child device:

1. Make a PUT request with the contents of the c8y-configuration-toml to: http://
   :80/tedge/file-transfer//c8y-configuration-plugin
2. Notify the parent device this was done by sending an MQTT message with the following topic and payload:

Topic: tedge/<CHILD-ID>/commands/res/config_snapshot
Payload (json string): { "status": None,  "path": "", "type":c8y-configuration-plugin, "reason": None} 

3. Restart the tedge-mapper c8y, or manually send:

Topic: [c8y/s/us/<child-id>] 
Payload: 114,c8y_UploadConfigFile,c8y_DownloadConfigFile

On the cloud you should see a new device created, with configuration management enabled.

child device config snapshot

this assumes the child device has already been bootstrapped.

1. From the cloud trigger a configuration snapshot for "file_a"
2. On receiving the following mqtt message from the parent device:

Topic: [tedge/<child-id>/commands/req/config_snapshot] 
Payload: {"url":"http://127.0.0.1:80/tedge/file-transfer/<child-id>/config_snapshot/<file-type>","path":"/path/to/file","type":"<file-type>"}

The child device should send an executing back:

Topic: [tedge/<child-id>/commands/res/config_snapshot]
Payload: {"status": "executing", "path": "/path/to/file", "type": "<file-type>", "reason": null}

3. Next the child device should use the URL above (from the config snapshot request payload) to PUT the file in the file transfer repository (http://ADDRESS:80/tedge/file-transfer//config_snapshot/)
4. After that the child device should send successful:

   Topic: [tedge/<child-id>/commands/res/config_snapshot]
   Payload: {"status": "successful", "path": "/path/to/file", "type": "<file-type>", "reason": null}

5. The configuration should be visible in the cloud, on the child-device overview, in the configuration management tab.

child device config update

this assumes the child device has already been bootstrapped.

1. From the cloud trigger a configuration update for "file_a"
2. On receiving the following mqtt message from the parent device:

Topic: [tedge/<child-id>/commands/req/config_update] 
Payload: {"url":"http://ADDRESS:80/tedge/file-transfer/<child-id>/config_snapshot/<file-type>","path":"/path/to/file","type":"<file-type>"}

The child device should send an executing back:

Topic: [tedge/<child-id>/commands/res/config_update]
Payload: {"status": "executing", "path": "/path/to/file", "type": "<file-type>", "reason": null}

3. Next the child device should use the URL above (from the config snapshot request payload) to GET the file in the file transfer repository (http://ADDRESS:80/tedge/file-transfer//config_snapshot/)
4. The child device should move the downloaded file to the "path" provided in step 2.
5. After that the child device should send successful:

   Topic: [tedge/<child-id>/commands/res/config_snapshot]
   Payload: {"status": "successful", "path": "/path/to/file", "type": "<file-type>", "reason": null}

6. The new configuration should be updated in the path on the child device.

[gligorisaev]

TEST CASE

Precondition:

Parent device:

1. Parent device is connected to c8y
2. Parent device has no child device connected
3. External MQTT bind address is set
   3. sudo tedge config set mqtt.external.bind_address <PARENT IP>
4. External MQTT port is set
   5. sudo tedge config set mqtt.external.port 1883
5. Reconnect Cumulocity
   7. sudo tedge disconnect c8y
   8. sudo tedge connect c8y
6. Start the c8y-configuration-plugin
   10. sudo systemctl start c8y-configuration-plugin.service

Child device

2. Moscquitto is installed on the child device
   1. sudo apt-get install mosquitto -y
   2. sudo apt-get install mosquitto mosquitto-clients -y

Test Steps:

On child device

1. Create configuration file for the child : printf 'files = [\n '\%3s' { path = '/home/pi/file_a', type = 'file_a' },\n ]\n' > c8y-configuration-plugin

Bootstrapping

2. Create config file on child device
   3. printf "files = [\n\t { path = '/home/pi/file_a', type = 'file_a' },\n ]\n" > c8y-configuration-plugin
3. PUT config file from child to the parent device curl -X PUT http://<PARENT IP>:80/tedge/file-transfer/<CHILD ID>/c8y-configuration-plugin \ --data-binary @- << EOF files = [ { path = '/home/pi/tedge/file_a', type = 'file_a' }, ] EOF`
4. Notify the parent device this was done by sending MQTT message mosquitto_pub -h <PARENT IP> -t "tedge/<CHILD ID>/commands/res/config_snapshot" -m '{ "status": null, "path": "", "type":"c8y-configuration-plugin", "reason": null}'
5. Send MQTT message to create child device on c8y (or restart tedge-mapper) mosquitto_pub -h <PARENT IP> -t "c8y/s/us/<CHILD ID>" -m "114,c8y_UploadConfigFile,c8y_DownloadConfigFile"

On Parent Device

1. Restart tedge mapper
   2. sudo systemctl restart tedge-mapper-c8y.service
2. Subscribe to listen
   3. tedge/<CHILD_ID>/commands/req/config_snapshot

On Cumulocity IoT

Child device config snapshot

Go to: Device Management > Devices > All Devices > Choose your parent device > Child devices

Confirm that child device MQTT Device <CHILD ID> is listed

1. Click on the MQTT Device <CHILD ID>
2. Click on menu item Configuration
3. Click on file_a that is located in the Configurations tab

4. Click on Get snapshot from device

   Result: Observe on parent device listener that following message is arrived:

   [c8y/s/ds] 526,<CHILD ID>,file_a

   and following message is sent to the child device

   [tedge/<CHILD ID>/commands/req/config_snapshot] {"url":"http://<PARENT IP>:80/tedge/file-transfer/<CHILD ID>/config_snapshot/file_a","path":"/home/pi/tedge/file_a","type":"file_a"}

On child device

1. Send MQTT executing message mosquitto_pub -h <PARENT IP> -t "tedge/${CHILD_ID}/commands/res/config_snapshot" -m '{"status": "executing", "path": "/home/pi", "type": "file_a", "reason": null}'
2. Send snapshot by putting the file to the transfer repository curl -X PUT -d "test of put" http://<PARENT IP>:80/tedge/file-transfer/<CHILD ID>/config_snapshot/file_a
3. Send MQTT successful message
   5. mosquitto_pub -h <PARENT IP> -t "tedge/<CHILD ID>/commands/res/config_snapshot" -m '{"status": "successful", "path": "/home/pi", "type": "file_a", "reason": null}'

Child device config update

1. On the Management tab choose Configuration repository and create new Configuration file_a
2. Go back to Child device configuration tab
3. Click on file_a that is located in the Configurations tab
4. Click on the newly created configuration file which appears in the Available supported configurations tab

5. Click on Send configuration to device

   Result: Result: Observe on parent device listener that following message is arrived:

   [c8y/s/ds] 524,<CHILD ID>,file_a

   and following message is sent to the child device

   [tedge/<CHILD ID>/commands/req/config_update] {"url":"http://<PARENT IP>:80/tedge/file-transfer/<CHILD ID>/config_snapshot/file_a","path":"/home/pi/tedge/file_a","type":"file_a"}

On child device

1. Send MQTT executing message mosquitto_pub -h <PARENT IP> -t "tedge/${CHILD_ID}/commands/res/config_update" -m '{"status": "executing", "path": "/home/pi", "type": "file_a", "reason": null}'
2. Send snapshot by putting the file to the transfer repository curl http://<PARENT IP>:80/tedge/file-transfer/<CHILD ID>/config_update/file_a
3. Send MQTT successful message
   5. mosquitto_pub -h <PARENT IP> -t "tedge/<CHILD ID>/commands/res/config_update" -m '{"status": "successful", "path": "/home/pi", "type": "file_a", "reason": null}'

[rina23q]

QA check passed, therefore close this issue.