Support for installing packages from custom repository

[mjj29]

Is your feature request related to a problem? Please describe. I would like to be able to use the software deployment repository to deploy packages from a custom apt repository. I have multiple interdependent packages with dependencies into the main repository, so I can't just upload a .deb file. apt install will work, but only if you've configured my custom repo first. Multiple software repository entries we're not convinced will necessarily be processed in the correct order to install them.

Describe the solution you'd like A single Software Repository entry which deploys the named package and all its dependencies from both the custom repo and the standard repository, in some fashion.

Describe alternatives you've considered Options I've thought of so far:

• URL in apt deployment package can be in a new form to specify the repo, eg:
  • deb https://downloads.apamacommunity.com/debian 10.15 nonfree
  • deb https://downloads.apamacommunity.com/debian 10.15 nonfree https://downloads.apamacommunity.com/debian/apama-repo-key.gpg (this one specifies the key to authenticate the repo, otherwise it would have to be marked trusted)
• URL in apt deployment package can point to a .sources file which specifies the repository, eg:
  • https://downloads.apamacommunity.com/debian/apama-10.15.sources _(you can see an example of that file here: https://svndae.apama.com/apama_installers/Apama/debian-apt/apama-10.15-internal.sources)_
  • Same issue with the repository key
• thin-edge.io distributes the Apama repository + key configured out of the box

If the URLs (in the .sources file) are HTTPS then having the apt plugin mark them as trusted, so they don't have to validate the GPG key is a little more reasonable, since they should be validated against the SSL PKI instead

Additional context

• Sample repository for testing: https://svndae.apama.com/apama_installers/Apama/debian-apt/
• Sample package to install: apama-thin-edge-support

[reubenmiller]

Thanks @mjj29 for the suggestion. We are indeed investigating different forms of packaging where one is include hosting the debian (.deb) files in a repository making the installable via apt-get install.

We are still evaluating the options as we would like to get it right the first time and not have to change again in a few months time.

[reubenmiller]

Duplicate #1560

[mjj29]

Hey, which ticket is this a duplicate of? I'd like to be able to follow the development of this feature

[reubenmiller]

It is #1560 ~#1506~ (I mentioned it when closing it). Idea is to get a better idea on what work is still required and having multiple tickets makes it a bit more confusing to track…hope that is ok

[mjj29]

This is 1506?

[reubenmiller]

Apologies, my fault. The correct ticket is #1560

[mjj29]

It's not quite the same as 1560 - that talks about the official thin-edge.io assets being in an apt repository - which could reasonably be implemented by manually configuring the repository before installing. However, this is to change the apt deployment plugin so that additional software can be installed from an apt repository that may be different to the manually configured one, through the cumulocity UI.

[reubenmiller]

Ah yes of course. For some reason I totally misinterpreted your ticket.

Though I’m fairly confident that this could be supported by use of the configuration plugin to manage the apt source lists under /etc/apt/sources.list.d/. As the apt plugin should pull in the required dependencies as long as the apt sources are configured before hand.

[reubenmiller]

And you can always create a meta Debian package which just references the packages you would like to install if you want that convenience factor

[mjj29]

This is what I did for the short term, but it's still two steps and still relies on the c8y software management always trying to install them in the right order.

One other related thing I've noticed is that right now you have to specify the full version of the package you would like, rather than 'whatever the latest fix is please' - which would seem better?

[reubenmiller]

Though there is the option to create a meta debian package (and there are easy to use tools out there like fpm which can be used to create debian packages). The meta package can then define the intended install order by using Pre-Depends fields etc.

One other related thing I've noticed is that right now you have to specify the full version of the package you would like, rather than 'whatever the latest fix is please' - which would seem better?

Yes I think a special version tag like latest would also be nice here. Technically if you send the operation using Cumulocity API you can send a blank version and the apt plugin will pick the latest available version, however the UI does not allow to specify a blank version so it is not very user friendly.

[reubenmiller]

@mjj29 I just create an issue to track the latest improvement idea. Could you please review it? https://github.com/thin-edge/thin-edge.io/issues/1594

[mjj29]

BTW, I don't think that pre-depends works when the first package defines the repository that contains the second package. The issue here is the order that c8y's software management tool will give the installation requests to the device, not the installation order of the packages

[reubenmiller]

Yes true I don't think the Pre-Depends will work in this case, as if the package is providing the apt repository config, then an apt-get update needs to be run before you can install a package from it (and this is impossible to run in the maintainer scripts due to the existing apt lock due to the current apt installation).

Normally this use-case would be solved by something like Cumulocity's device profile handling. This would provide the possibility to combine configuration and software packages into one operation where the operation where you could define that configuration files should be installed before software packages.

[reubenmiller]

But current setup it would require you to apply the apt repository configuration first via apt or a configuration, then installing the package. Is this ok for now?