didier-wenzek
commented
3 months ago The PR introducing support for log and config operation workflows, also introduced a regression bypassing file cache for config update.
- Previously, configuration updates were first downloaded on the gateway device to be locally available for the child devices.
- Now, this preparation step is skipped and the child devices are using unchanged the url provided by the config update command:
- i.e. a local url to c8y http proxy when the device is connected to Cumulocity
- a remote url to the cloud resource for the other clouds.
- In other words, currently, the configuration of a child device can only be updated from Cumulocity.
We must re-introduce this ability:
- to cache on the gateway device a file downloaded from the cloud
- and to make this file locally available over HTTPS to child devices, containers and services
- not only for configuration files (as previously done)
- but for all kind of resources (configuration files, software packages, firmware updates).
Proposal
Introduce a new file_cache operation:
- triggered from any device or service on
te/<entity-id>/cmd/file_cache/<request-id>with a payload{ "status": "init", "remote_url": "<remote-url>" } - handled on the main device by the agent which downloads the file
and makes it available with a local url that is returned to the requester
{ "status": "successful", "remote_url": "<remote-url>", "local_url": "<local-url>" }.
Pros:
- The
file_cacheoperation can be triggered as a sub-operation from a workflow - There is no fundamental changes neither on the workflow system nor on the file cache actor. The latter was specialized on configuration updates, it has simply to be extended to address more usages.
Cons:
- This is not a true command: there is no capability message and its execution model is unusual (handled by the main agent whatever the requesting device).
- When should the cache be invalidated? Using a TTL? Note, that despite its name, the file cache actor was caching nothing simply downloading each file on each request.
Alternatives:
- Make this operation a plain operation only available on the main device.
- In that case, the workflow mechanism must be extended with the ability to trigger an operation on another device
- Use a request / response pattern with two topics
te/device/main///req/file-cacheandte/device/main///res/file-cache- The workflow mechanism must be extended to trigger such operation.
- Extend the file transfer service with a POST command to upload a remote resource and to make it available locally.
- But then how to tell the requester that the resource has been fully downloaded and is locally available?
- By keeping the connection open till ready?
- With a combination of
202 (Accepted)and polling? - With a combination of
202 (Accepted)and MQTT messaging ?
reubenmiller
Is your feature improvement request related to a problem? Please describe.
Installing firmware/software/configuration on child devices requires the child device to download the given url directly from the internet, or via the c8y proxy service (which requires internet). Both of these downloads means that the child device needs to support retries and resuming of the download which is sometimes not feasible for devices such as microcontrollers.
Describe the solution you'd like
thin-edge.io should support a mechanism where the child device can request thin-edge.io to download and cache the artifact so that the child device only has to download the artifact via a local cache.
The caching mechanism should be supported on any of the commands (e.g. firmware/software/configuration).
Describe alternatives you've considered
Additional context