Create a central repository for SPM

[johnkeates]

Create a central repository for SPM to make it actually useful.

There currently is no usable central SPM repository to use, this makes SPM actually less useful than maintaining a mess of git submodules in a git repository to manage formulas. At this moment, either home-cooked formulas or formulas from github.com/saltstack-formulas are the best source, and SPM plays no role in it. No formula on the saltstack-formulas repo has any SPM metadata (as far as I could find) and there is no usable dependency management as a result.

Now that the SPM format is defined, it would be helpful for something like a central PECL-style or Packagist-style repository to be available. Adding a salt-provided community-driven central repository would also help with adopting the SPM format.

[ssgward]

@johnkeates - thanks for the feedback and ideas.

[jfindlay]

Ping @techhat, @UtahDave.

[johnkeates]

I imagine this as a relatively simple webservice where an author can authenticate using existing providers for simplicity (like GitHub, Bitbucket or something like that), connect a repository or manually upload a zip with a new SPM package. It shouldn't be that hard to clone one of the existing concepts. The issue would of course be: who's going to build it, who's going to host it, and who's going to pay for the bandwidth. I'm happy to help, but sadly, I don't have a spare server or free bandwidth to provide, so it's going to be engineering only from my side if any help is practical from outsiders.

[johnkeates]

If I were to start implementing this thing, where do I PR it?

[ssgward]

@johnkeates - Add a PR in this same repo: saltstack/salt.

[themalkolm]

Why not instead package *.spm packages as *.rpm, *.deb etc. and use already existing infrastructure that almost for sure exists in every company?

[johnkeates]

Because they are complex packages, while packages like the ones used by nodejs and php (composer) are fairly simple. On top of that, rpm/deb is designed for end-user software, or software in general, whereas spm (and npm and composer) are for 'code' more than a finished product.

Furthermore, windows and OSX don't do rpm/deb, neither does any BSD/Unix. We'd have to make .pkg's, msi's, pbi's, snappy app packages, opkg's for embedded devices etc...

[themalkolm]

No, I don't suggest to replace spm with rpm/deb/etc. This is definitely a bad idea. Just simply wrap it in rpm or another native packages. This way you use same environment and don't need a separate web service, hoster and somebody to pay for all this.

There is no need to go deep in different package systemst as the only thing you need in the end is to deliver a singe spm file mongodb.formula.spm -> /var/cache/spm/packages/mongodb-formula.spm and invoke a post install script spm local install /var/cache/spm/packages/mongodb-formula.spm later. On package remove simply invoke spm local remove mongodb-formula.

It should be fairly easy to implement for the most popular systems, right?

[johnkeates]

I suppose for most popular systems that could work, but that doesn't solve the dependency resolution problem. The only reason SPM would work and could get traction is if you can do something like spm install graphite and it would pull in and dependencies it needs, instead of having to do spm install graphite postgres memcached redis nginx.

Setting up a repo server for SPM isn't hard, and it's not like composer and npm were that hard to setup ;-) Even NuGet for C# works, and PECL, and PEAR! (and CPAN, and all those others).

Another reason would be central publication of formulas, and the option for comments, votes on security and integrity, and maintainers that might need to change the upstream source.

Say you have a formula you'd like to share, you just dump it on github and maybe link it to saltstack-formulas, and that's it. Not the nicest way to do this, and the lack of SPM usage makes resolving dependencies or creating meta-formulas that just pull in a selection of formulas that you need to set something up really hard.

With an SPM "server" one could set a system up locally, or maybe SaltStack would host a public one, or possibly we'd just use jekyll and host it on github (which would make github sad, as happened with CocoaPods). It's not as much that it needs to be a hosted catch-all thing, but right now there is nothing there at all, and that's just not working.

Even just for internal use, I need a webinterface where I can do this:

formula (on github perhaps) -> Add to SPM server and point to github for upstream updates -> define any dependencies -> click publish -> done

on workstation and/or master: spm install package (thing downloads package and puts it in the right spot) and then I can use the package.

I already run internal apt repos (I tend to stay away from RPM systems), and I don't really feel like building an infrastructure like I just described but then also having to interface it on top of that repository.

[themalkolm]

Maybe we come from different point of views and experience. Personally I don't see spm doing anything more that simply install, configure and remove salt formulas. It looks way too young to rely on as well e.g. I couldn't find a way to list all packages spm believes it installed.

So for us the low hanging fruit is to use only salt-related logic from spm but do everything else with already well-known tools. In our case it is rpm, yum etc.

We just propagate dependencies (from FORMULA file) up to rpm package and let yum it figure requirements and install all missing salt formulas. As a bonus minimum_version is converted to something like requires: salt >= 2016.3.0 and this way we can be sure that yum won't allow us to install something that was not written for this salt version or even upgrade salt and break all formulas.

Maybe the difference is that with this approach you can get this right here right now while spm takes very moderate pace in development (it seems) and will take a while to become actually useful.

[themalkolm]

Just to be clear, I'm aware that this works for us but isn't that useful for other users. Just describing what we've got.

[techhat]

Dependency resolution and enforcement has been added and should work (ref #31469).

As @themalkolm states, SPM is designed only to install, manage, and remove Salt-related files.

@johnkeates and @themalkolm, you both raise some important points, and I would love for you both to be more involved in the development of SPM.

[techhat]

@johnkeates I should also mention that I am currently working on a graphical front end for managing SPM repositories. @ssgward, it will not be part of the main Salt distribution, as the vast majority of Salt users will have no need for it. However, for those that do, it will be crucial. I will post an update here when it is available.

[johnkeates]

Good to know others are thinking and working on this thing. I see that Salt's SPM target was to simply supply a standard for formula installation based on a simple format and nothing else, but I thought the fact that it has channel/repo code in it meant someone already had the central repository in mind.

Separate from who's doing what and what the best way to go about it: I do wonder how we're going to create a central 'thing' to store formulas in. I don't see current saltstack-formuals contributors making OS-specific packages, or any packages at all. At best we can expect a FORMULA file with basic data on the formula itself and that'd be it.

Manually making packages isn't going to work out very well, and neither is asking people to maintain other people's packages based on sources they don't control.

Taking this back to a basic form where you have a publicly accessible git repository somewhere, you'd need a way to assure:

• quality control
• available dependencies
• maintenance
• stuff gets put into a SPM package

Then, for each distribution you'd need a package repository, to accommodate it's preferred style, plus maybe a tarball server where you can just download the SPM packages directly.

If we take a look at a few other configuration management tools, they have their own repositories;

https://supermarket.chef.io/cookbooks https://forge.puppet.com https://galaxy.ansible.com

Even Microsoft has something: https://gallery.technet.microsoft.com/scriptcenter/

If we take a look at a few domain-specific repositories for other systems, they have:

https://www.npmjs.com https://packagist.org http://search.cpan.org https://pypi.python.org/pypi https://pecl.php.net/packages.php

They all work on all systems where their OS-specific counterparts were installed.

If we take a look at what Salt itself supports:

DEBIAN AND DERIVATIVES Debian GNU/Linux 7/8 Linux Mint Debian Edition 1 (based on Debian 8) Kali Linux 1.0 (based on Debian 7)

UBUNTU AND DERIVATIVES (probably just supported by Debian packages) Elementary OS 0.2 (based on Ubuntu 12.04) Linaro 12.04 Linux Mint 13/14/16/17 Trisquel GNU/Linux 6 (based on Ubuntu 12.04) Ubuntu 10.x/11.x/12.x/13.x/14.x/15.04

RED HAT FAMILY Amazon Linux 2012.09/2013.03/2013.09/2014.03/2014.09 CentOS 5/6/7 Fedora 17/18/20/21/22 Oracle Linux 5/6/7 Red Hat Enterprise Linux 5/6/7 Scientific Linux 5/6/7

SUSE FAMILY openSUSE 12/13 openSUSE Leap 42 openSUSE Tumbleweed 2015 SUSE Linux Enterprise Server 11 SP1/11 SP2/11 SP3/12

OTHER LINUX DISTROS Arch Linux Gentoo

BSD: OpenBSD (pip installation) FreeBSD 9/10/11

SunOS: SmartOS

That's a lot of OS-specific packages we'd need to supply! Of course, this is possible if we have a method of automatically converting sources with a FORMULA file into an SPM package inside a bistro-specific package. But then you'd also have to host separate repositories for all of them, or do a whole lot more work to comply with all the distro-specific rules to be added into their repo's. And updates would be slow, unsupported or you'd get stuck om some volatile list where stuff like anti-virus software stores it's definitions.

I'm not saying that there is no way it could work, but adopting an existing domain-specific packaging system seems to be a lot more sensible than sticking SPM inside a package.

Aside form the whole packaging deal; what about a Salt-branded website where you can browse formulas? That would be a good thing in general, even if it only has links to github repos or SPM files.

So far, to me, it looks like a few things will be needed to get this working:

• server side component where formulas are published in search-/browsable lists
• client side component that gets the formulas and dependencies like a web app formula that needs a db formula and a runtime formula to work

[techhat]

I'm actually glad that none of the saltstack-formulas are intentionally specific to just one package. A number of them do explicitly support multiple distributions, usually RedHat and Ubuntu at the very least. And since SPM is platform independent, we have ability to create a single Apache package (for instance) that can be used for multiple OSes.

[johnkeates]

@techhat that's pretty much why to me it wouldn't make sense to then deliver packages using platform-specific packaging tools. I get that it's easier if you happen to already use it, so some people might consider it, but it doesn't seem to fit in line with what Salt has been doing so far.

I've seen people trying to implement a configuration package for a complete ELK stack in a single formula, but that's just a bad idea. If someone were to want an easy way to setup something like that, separate Elasticsearch, Logstash and Kibana would be better, and then maybe a meta-formula that just depends on those three and maybe has a few strings in a defaults.yml file to get it going with sane defaults.

I do hope more people come in to this conversation and add more perspective or points of view on this, as I'm sure no single view is the best and the only way to get a nice system that works well enough for everyone is to find the common denominator(s).

[themalkolm]

So why not distribute everything via, say, pypi server? People have it installed in their companies, they know how to work with it, every python developer knows how to browse, search and work with it. A lot of known issues and solutions to them. We even have pip and other tools to work with it right out of the box. Win?

Where the urge to write the very own salt-based solution comes from? Well, except the obvious vendor lock-in idea. Just curious.

[johnkeates]

Well, that's what I was thinking about. Adopting any of the existing systems as I wrote. pypi makes sense, since we're in Python-land and it's bistro-agnostic. NPM's format or even Composer's thingy would work as well.

[techhat]

Not every Salt user is also a Python user. In fact, these days most don't seem to be.

The intention was to create a format that was easily malleable without having to shoehorn our needs into somebody else's solution.

FWIW, I am completely and utterly against vendor lock-in.

[johnkeates]

In that case, we can't really use any developer-oriented package management system, but no distro-agnostic non-developer-oriented comes to mind right now.

Maybe checking for any underlying universal package management libraries that could be added to SPM's CLI for channel/repo would be the next best thing, that way we won't be reinventing the wheel but still having domain-specific package sources.

[themalkolm]

Well, if we think about users then I don't think they really care what magic commands to type. Not like it is hard to make a nice cli wrapping python/pip/ruby/.. magic. Some people use vagrant without even realizing that it actually a ruby code they edit in Vagrantfile.

Maybe if we want this to fly we should really focus on devs and future package maintainers. What do you think?

P.S. Yes, all these mentioned package systems already solved a lot of issues spm is guaranteed to face like db, dependencies etc. Some even have solutions as standalone tools e.g. libsolv, librepo etc.

[techhat]

@themalkolm by default the package database is stored as /var/cache/salt/spm/packages.db. It is in sqlite3 format. However, the package database uses a loader system, so you can break it out into whatever database format you feel is appropriate. As mentioned above, we also already have dependency resolution added.

Another note here is that SPM also uses a loader system for installing files. By default files are installed on the local filesystem, but you have the ability to write, for instance, a module that installs files into a data store that is already being served using one of the external filesystem modules. I don't know of any other package managers that support that kind of modularity.

[themalkolm]

@techhat aha, I somehow skipped over this concept. Thank you for pointing this out. Quite powerful idea, indeed. Let me read again SPM docs and come back later.

[techhat]

@themalkolm I'm sure it's not documented quite as well as it should be. Should you file new issues to point out the deficiencies, I would be happy to fill them in.

[themalkolm]

@techhat Now I recon why I skipped this part. It is because this page is impossible to read https://docs.saltstack.com/en/develop/topics/spm/dev.html :)

I think it will be better to translate this human descriptions to code snippets like the one in the bottom of the page (hash_file). It is for devs, right? So, compare:

INFO()

Return information for a package. This generally consists of the information that is stored in the FORMULA file in the package.

The arguments that are passed in, in order, are package (required) and conn (optional).

package is the name of the package, as specified in the FORMULA. conn is the connection object returned from init().

def info(package, conn=None):
    """
    Return information for a package. This generally consists of the information
    that is stored in the FORMULA file in the package.

    Args:
        package (str): Name of the package, as specified in the FORMULA.
        conn (Optional[any]): Connection object returned from init(). Defaults to None.

    Returns:
        dict: Package info
    """
    ...

I find it much easier to read, understand and I immediately wonder what should this function return if this package doesn't exist? None? Empty dict? Some default value? Raise something?

[ghostsquad]

👍 on having a salt-hosted repository for formulas (like docker hub).

[johnkeates]

I have this bad feeling that Salt will implement this for their top secret Salt Enterprise code and we'll have to make one ourselves out-of-tree :(

[DanielFallon]

@themalkolm @johnkeates I was looking at ways to do this in the last week, and it really seems like for more complex reusable formulas it makes more sense to place this in pypi with namespaced package names.

to be clear this is a proposal to generally leave behind spm conceptually

most other python projects use pypi/pip to handle dependencies, and despite its shortfalls, consistency is important within a development community.

If we use pip to handle dependencies: we solve:

• version dependency
• public repository
• any maintenance concerns for spm

we do not solve:

• multiple versions of the same package (as pip does not solve this)

My Proposal

A file server backend that loads a list of python modules (specified in master config) and uses their contents to expose files to the file server

A simple import could be created to convert a file server/FORMULA root into a python package. This could be used to generate scaffolding for formulas.

sample package:

salt-formula-apache/
    salt-formula-apache/
        salt/
            apache/
                init.sls
                ...
        init.py
        configuration.py
    setup.py

Question

Is there some reason that this proposal doesn't make sense, and what pitfalls do others see with this approach (aside from learning curve due to the difficulties of python packaging)

After I have feedback, I have budget and time to complete something like this as my company is currently trying to figure out how to deploy changing infrastructure code for highly custom deployments (read can't fit in a docker container)

and I'd be happy to work on this, but I want to make sure that my approach isn't misguided

References:

• django-packaging
  • sphinx extensions

[themalkolm]

@DanielFallon in the end we simply write rpm packages for all formulas. For custom behaviour during installation there are yum plugins. Made no sense for us to continue with spm.

[groggemans]

@sjorge @Ch3LL This is what I was talking about this afternoon when compared to puppet modules or ansible roles. (Although when I first looked at Salt, SPM didn't exist, and it seems to go in the right direction)

I share a lot of @johnkeates's ideas/concerns. If it's part of enterprise (?), I do hope it comes to open-source. I think it could give a big boost to the Salt adoption.

[Ch3LL]

I spoke offline internally with the team to get some answers and this is something that we are wanting to do open source wise but is not currently being worked on due to other higher priority task items. But its something that we definitely want to do. :)

[oogali]

It's been briefly discussed in this thread, but there absolutely needs to be strong guarantees around package versions.

Every package manager referenced in this thread has significant complexity around declaring versions, upgrading versions (by patch level, minor, etc), and allowing/disallowing multiple versions to be installed.

Something which hasn't been discussed thoroughly is what versions of a package a SaltStack administrator wishes to have installed. I may want 0.1.2, Bob may want 1.0.*, and Alice may want latest.

All package managers wrestle with this, and the result in every case has been to create yet another file that declares dependencies and the ideal versions (Ruby: Gemfile; Python: requirements.txt, setup.py, Pipfile; NPM: package.json; PHP: composer.json).

I like @DanielFallon's proposal as:

• it reuses an existing package manager that has already dealt with and solved a lot of the problems of package management, distribution, and version selection (or at least, has a publicly well-known behavior pattern),
• already has its own distribution mechanism so Salt doesn't have to maintain a public SPM infrasturcture,
• already has its own supporting foundation behind it so Salt doesn't have to "tax" enterprise users or solicit donations to support SPM,
• is cross-platform (pip works on Linux, Windows, OS X, etc) so less resources have to be allocated to development and testing of SPM,
• is closest to Salt packages (Python) than OS distributions (RPM/DEB/MSI/etc),
• and doesn't require administrators to maintain yet another package manager (or require SaltStack to break SPM out into its own independent package which is separate from salt-common).

[viq]

While talking about an SPM repository, central or not, there's one potentially quite important thing that most of package managers do, where pypi (and currently SPM) don't seem to - signing packages to prove where they came from and that they weren't modified. Are there any ideas around that?

[johnkeates]

So this is still open... I've seen some work on related tickets, but does that mean this is idle or simply not going to happen?

[Ch3LL]

as it currently sits it is simply idle and not currently being worked on due to other higher priorities.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[dafyddj]

Not stale.

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[mtusseau]

I agree with @dafyddj. I also plan to have a self-hosted SPM server, based on POC given in #43318, but probably with Sonatype Nexus repository.

[johnkeates]

Dave Verwer has created a website that catalogs many SPM frameworks with helpful info like license, versions available, and how long ago the last update was.

The Swift Package Index

That seems to be a Swift repo, not SaltStack.