ncm-spma: More package managers

[jrha]

So now we have a pattern for adding support for other package managers, the obvious thought is to build support for apt.

However, during a talk at FOSDEM yesterday it occurred to me that there are more and more package managers on our systems, some are good, others less so.

pip is very useful, if sometimes messy. npm seems to be one of the better ones.

Both of these:

• Supply software that is difficult to obtain as RPMs
• Support version pinning.

How would we feel about supporting them? How ugly would having more than one active package manager be?

[piojo-zz]

How would we feel about supporting them?

I'm happy to integrate any package manager that does all of the following:

• Supports installing software.
• Supports removing software.
• Supports upgrading software.
• Supports downgrading software.
• Supports querying what software is installed.
• Supports pinning versions.

How ugly would having more than one active package manager be?

If backwards-compatibility is a strict requirement, it would suck. In that case it would be better to have an ncm-npm and put NPM packages in that subtree.

If we could break backwards-compatibility, we might turn /software/packages/spma/packagers into a list, and have the component dispatch them. Then, use /software/npm/packages/..., /software/pip/packages... and leave /software/packages for the first element in the packagers list. Then, instead of a framework, ncm-spma would become a dispatcher.

Or something more carefully designed.

[msmark]

I think ncm-spma is already a dispatcher. Note I couldn't use /software/packages for the IPS module. Instead I used the paths /software/requests, /software/catalogues and /software/uninstall. But rather than have those as hard-coded paths in the module, they are exposed in the spma schema, as described in the IPS documentation:

       /software/components/spma/pkgpaths : string []
           Contains a list of resource paths where catalogues and individual
           package requests are located.  Should be set to:

               list("/software/catalogues", "/software/requests");

       /software/components/spma/uninstpaths : string []
           Contains a list of resource paths where packages to uninstall are
           located.  Should be set to:

               list("/software/uninstall");

The distinction between /software/catalogues and /software/requests is also described in the IPS documentation.

So I think all future packagers might also want it to be configurable as to where the package information resides.

I also slightly redefined the meaning of /software/components/spma/run:

       /software/components/spma/run ? string
           Set to yes to allow this module to launch spma-run --execute to
           make immediate changes to the new boot environment.  If set to no
           or omitted, this module prepares and validates the changes only,
           but does not perform any updates, it will be the responsibility of
           an external process to launch spma-run --execute in this case.

I added some new stuff to the schema that is currently only used by IPS but that might be used by other package managers too:

/software/components/spma/cmdfile
/software/components/spma/flagfile

Note I also have ips-specific things in the schema, for example:

/software/components/spma/ips/bename
/software/components/spma/ips/rejectidr
/software/components/spma/ips/freeze

We will want to see some changes in the Yum provider at some point this year to get it to honour /software/components/spma/run = no treating it the same way as the IPS module does, to support the distinction between catalogues and requests as described above, to support a list of packages to uninstall as well as a list to install, and make use of /software/components/spma/pkgpaths, /software/components/spma/uninstpaths, /software/components/spma/cmdfile and /software/components/spma/flagfile. We will also want to modify the spma-run tool to work with Yum as well as IPS.

Can we standardise on some of this work, and suggest that new package providers also support these features?

[msmark]

Re backwards compatibility, I think /software/components/spma/packager (singular) should remain a string. If you wanted to introduce /software/components/spma/packagers (plural) as a list you could do that as long as you still allowed the original form. I don't think backwards compatibility sucks, I think it's a necessity and it's not an ugly one.

[samary]

Hi,

I use pip a lot here, very useful to manage python dependencies. A spma pip backend will definitely help me to get rid of my ugly custom scripts ! Which prefix will you use to define them ?

[stdweird]

@samary make rpms and put them in a repo. python software distribution tools are a nightmare anyway. what packages are you looking for?

[samary]

Not a specific one but if we could have it in SPMA it would be wonderful. For example to install speedtest-cli : pip install speedtest-cli. I didn't found a rpm.

To get the latest version of some tools not packaged yet, it can be also useful IMHO If I am the only one asking this, just let it go. I'll use the pip list and loop through and install them via a file-copy.

[stdweird]

@samary no, you really don't want anything else then rpms. (unless you understand the magic of the .pth files, and in that case, you can explain me ;) to make an rpm, we use scripts. eg in your case:

yum install rpm-build rpmrebuild
./buildpythonrpmfromsetup.sh speedtest-cli

https://github.ugent.be/stdweird/quattor-SCDB-ugent/blob/master/othertools/buildpythonrpmfromsetup.sh

(feel free to get rid of the .ug tag; we just use it for all rpms we build ourself)

[samary]

Hi,

Thanks a lot for the advice and the script !

I will have a look and try to build some rpm from python and give some feedback if needed.

Cheers

[jrha]

I've been thinking about this some more and think I have decided that non-core package managers should have a different component and not be mixed up with spma.

[jouvin]

I was about to say the same after reading this discussion this morning. I understand that this proposed architecture may be interesting in principle but I am not convinced we really have a solid use case for it. In particular, I am not sure (but I am not an expert!) that pip has all the features we require for using it as a ncm-spma backend where it is certainly possible to write a ncm-pip if we need it. For me the main reasons to have it as a ncm-spma backend are:

• Avoid unnecessary code duplication: not clear how much we may save in this case
• Have it as a dependency for other components included into the packager "metadependency" that is spma today. Do we have any use case requiring it?

Another issue was introduce by @msmark during the discussion that we should probably track down as a separate issue: noaction flag action in the case of SPMA. I think it deserves a real discussion, both on the proposal itself (that looks interesting to me) and on the implementation based on reintroducing spma-run.. Mark, would you take care of creating a separate issue for this?

[jrha]

Closing this, will create issues for anything specific that came out of it.