Design: Mixin Distribution

carolynvs commented 5 years ago

See #151 for how mixins are installed.

Right now we are handling distribution through install scripts. I want to get to the point where the porter cli can install mixins from external sources:

$ porter mixin install helm --source https://deislabs.blob.core.windows.net/porter/mixins/helm/v0.1.0/
downloaded helm-darwin-amd64
downloaded helm-linux-amd64
installed helm mixin to ~/.porter/mixins/helm@v0.1.0

The above is just an initial idea of what I hope to see. Essentially I want to be able to install mixins from anywhere and have porter be aware of the version.

I'm talking with @radu-matei and @bacongobbler about implementing this with libraries from https://gofi.sh.

carolynvs commented 5 years ago

It would be really useful if editors could implement a quick fix like "Find and install this mixin" when the manifest specifies a mixin not present on the local machine. Honestly porter may need something like that as well (so that if you authored on one machine and moved to another, you could quickly install your mixins).

We need to think about how we could either support this or make really good error messages so people knew how to find the missing mixins themselves. Maybe there is gofish magic that would help make this possible?

bacongobbler commented 5 years ago

perhaps a quick check if it's installed, and if not, install it?

The code in .Installed() is really brittle has a few edge cases that I can help explain if you'd like to discuss further. That does sound like a good use case for the gofish libraries however!

carolynvs commented 5 years ago

Sweet! Sounds like 🐠 will be able to get us pretty far.

carolynvs commented 5 years ago

These are my original, unedited notes. I'll do another pass and make them pretty / understandable to others beyond the 4 who were present tomorrow.

manifest

mixins:

name: helm@2 repo: https://github.com/deislabs/porter-mixins.git
name: terraform

suggestion: keep the fishfood lua in the same repo as the mixin (see helm plugin model)
- check out the mixin repo at the requested tag
- hard to do with shallow clones and do an upgrade without expensive recloning
- no searchability
alt method: use branches to identify older versions of mixins that you can pin to

if they leave off the repo, we will assume it's the default fish food from the config file

porter home

porter needs a config file I think

repositories:
  - name: deislabs
    url: https://github.com/deislabs/porter-mixins.git
  - name: azure
    url: https://github.com/azure/porter-mixins.git

unpack fishfood into: PORTER_HOME/mixins/repos/deislabs/helm.lua

describes how to install the helm mixin

install mixins into same location as before PORTER_HOME/mixins/helm/helm

Management commands

$ porter mixins list exec helm azure

$ porter mixin install helm --repo deislabs

$ porter mixin-source install https://github.com/deislabs/porter-mixins.git

$ porter mixin-source uninstall deislabs $ porter mixin-source list deislabs https://github.com/deislabs/porter-mixins.git azure https://github.com/azure/porter-mixins.git ## Action Items * terminology: get an idea of the gofish api: write a separate library with generic names porter and gofish can use it (fisher) * agree on architecture and use cases (above) * agree on versioning and how to pin on old versions Carolyn will make issues and split up the work for this so that we can get on it

carolynvs commented 5 years ago

This is the second round of notes on what the mixin commands will look like:

Mixin URL Format

Individual Mixin URL: URL/VERSION/MIXIN-GOOS-GOARCH[OPTIONAL_FILE_EXTENSION] Useful for when someone doesn't want to bother with making a feed, and just wants to share a single mixin. This works out-of-the-box with uploading binaries to github releases.
ATOM FEED: URL/atom.xml Perfect for someone who has many mixins, like deislabs. Lets us share a bunch and enable people to keep up to date with them.

Install

$ porter mixin install terraform --url https://github.com/deislabs/porter-terraform/releases/downloads
installing latest

// install latest by default
// remember that we installed it from this url so that later we can upgrade

$ porter mixin install terraform --version 1.0.0 --url https://github.com/deislabs/porter-terraform/releases/downloads
install 1.0.0

$ porter mixin install terraform --version canary --url https://github.com/deislabs/porter-terraform/releases/downloads
installing canary

$ porter mixin install helm --feed-url https://porter.sh/mixins/atom.xml
installing latest

Feeds

$ porter mixin subscribe deislabs --feed-url https://porter.sh/mixins/atom.xml
subscribing to 5 mixins from the deislabs feed
    helm
    exec*
    azure
    kubernetes
    docker

Found one mixin (flagged above) that was already installed and will now be managed
by this subscription.

$ porter mixin install helm [--feed deislabs]
installing the helm mixin from the deislabs subscription

// you can leave off --feed, it's optional, but you can specify if you are picky or there's overlap

$ porter mixin install --feed deislabs
installing all mixins from the deislabs feed
    azure@1.1.1
    kubernetes@0.1.0
    docker@0.1.1

$ porter mixin upgrade --feed deislabs
upgrading all mixins from the deislabs feed
    azure@1.1.1 -> azure@1.2.0

Upgrade

$ porter mixin upgrade --all
upgrading all mixins 
    blippy@0.1 -> blippy@0.2

$ porter mixin upgrade docker 
exec is available on multiple subscription feeds: deislabs, docker
edit the command below to specify the feed, and then run it

    porter mixin upgrade docker --feed TODO

$ porter mixin upgrade azure --version canary

$ porter mixin unsubscribe deislabs [--uninstall]
unsubscribed from the deislabs feed. The following mixins used to be managed
by this subscription and are still installed:
    helm
    exec

List

$ porter mixins list

MIXIN       VERSION                     SOURCE
exec        0.4.0-ralpha.2+flirtini     deislabs
blippy      0.1.1                       https://example.com/blippy
docker      0.2.0-beta.1                https://github.com/docker/porter-mixin/releases/downloads

$ porter mixin feeds list

FEED        URL
deislabs    https://porter.sh/deislabs/atom.xml

// this has mixin feed subscribe/unsubscribe aliased to mixin subscribe/unsubscribe

$ porter mixins list --feed deislabs -o json
// filter list of mixins by a particular feed

Mapping mixins from the manifest

When someone specifies a mixin without a source, porter will use whatever version is installed, and if it's not found run porter mixin install helm. So it will rely on the configuration of the subscriptions, and not be picky about what it uses. Otherwise it will use the values from the manifest to run a more specific install, such as porter mixin install docker --version 0.1 --url https://github.com/docker/porter-mixin/releases/download. It ensures that if the mixin was already installed, that it not only has the right version but came from the same source.

mixins:
 - helm
 - name: docker
   version: 0.1
   url: https://github.com/docker/porter-mixin/releases/download

vdice commented 5 years ago

This looks great to me 👍

carolynvs commented 3 years ago

We are going in a different direction now with mixins, and want to install per bundle instead of globally. If there are any ideas from this that aren't in our two mixin proposals, let's incorporate them into the PEPs there.

https://github.com/getporter/proposals/pull/7 https://github.com/getporter/porter/discussions/1499

getporter / porter