salt-call --local with gitfs

[aboe76]

I would like to propose the gitfs support for salt-call.

This would enable a lot of cool stuff: like easier deployment from saltstack-formulas using docker.io or packer.io without the overhead of a complete salt-master installation.

Requirements as I think of is that salt-call needs to parse /etc/salt/master.d/ or /etc/salt/minion.d/ files for correct information about what gitfs remotes there are..

[basepi]

This would be cool, thanks for the suggestion!

[mafrosis]

+1 for this. I started looking at it today - but have the feeling I'll do more harm than good hacking it in! Pointers on where to start are very welcome.

[ruimarinho]

I was about to write about this, but I'm glad someone found a need for it too! I genuinely thought salt would simply read a gitfs remote under /etc/salt/minion ;)

[F30]

+1

[pidah]

+1

[aromanovich]

+1

[ryan-lane]

+1 I could definitely use this for vagrant + masterless salt + formulas.

[ruimarinho]

I've found that salting a salt master is actually a very good alternative to salt-call --local if you accept the overhead of running another daemon locally.

[diegows]

+1 and I agree with @ryan-lane

[JackLeo]

+1

[gitfrage]

+1

[etuttle]

+1

[ahdinosaur]

+1

[jsuchal]

+1

[amahon]

+1

[DeTeam]

+1

[matthias-ollig]

+1

[mwildehahn]

+1

[johnnadratowski]

+1

[gepoch]

This would also bring support for using state.orch on purely gitfs backed systems. Orchestration files in a git repo just make sense.

[basepi]

@gepoch What's stopping you from using gitfs with state.orch now?

[gepoch]

At the moment (in 2014.1.4) salt-run state.orch some.state is giving me this:

salt:
    Data failed to compile:
----------
    No matching sls found for 'some.state' in env 'base'

I can access all of the formulae in the same git repo with no problem. Looking at the source (although this may have been a few patch versions ago) seemed to suggest that it wasn't capable of using the gitfs backend, but I could definitely be wrong about that.

It should just be a matter of configuring the gits_remotes to point to the right git repo, right?

[basepi]

Theoretically, yes. And the base environment is automatically mapped to the master branch of your git repo.

I should note that state.orchestrate has some pretty serious deficiencies in the 2014.1 branch -- there have been a lot of fixes and additions in the develop branch, so you may find that your issue has been fixed when we put out the release candidate for the next feature release. (In a week or two)

[gepoch]

Yeah, I'm wise to the master-base mapping and that's already okay.

I'm guessing it's in the develop code. Anyways, thanks very much! I'm looking forward to it :)

[joehoyle]

+1!

[davidkarlsen]

+1! This would be great for vagrant/docker/masterless salt boxes.

[cooncesean]

@davidkarlsen @ryan-lane What's the work around you two are using for a masterless Vagrant setup? Copy/paste the needed formulae into your local environment?

[ryan-lane]

For vagrant I used master/minion, which kind of sucks since I'm aiming for masterless in production. Note that I'm mostly using gitfs to make it easier for devs to quickly test things from formulas.

[davidkarlsen]

Both options would be possible. Either do checkouts to a local dir and include this in a masterless, or set it up as master/slave on and use gitfs. I´d probably go for the latter.

[mafrosis]

FWIW a work around for this in masterless setup is to use git submodules to add to the necessary states.

[DeTeam]

@mafrosis still ugly, though..

[mafrosis]

:smile_cat: you're not wrong... but it's easily portable to a master/minion setup and easily replaced when this feature gets in there.

[DeTeam]

@mafrosis true, wondering if it's an easy configuration task while working with Vagrant (:

[mafrosis]

I setup a Vagrant sync_folder which exposes all my salt states to the minion. Adding a git submodule to this is pretty straight forward. Take a look here, if it helps:

https://github.com/mafrosis/inform/tree/master/config/salty-vagrant

[DeTeam]

Oh, nice tip! Submodules are fine in that case. Thanks!

[mlister2006]

+1 for salt-call --local with gitfs.

[ProjectUn1c0rn]

If it can help as a temporary solution, we used this "hack" for our bootstrap script : https://github.com/ProjectUn1c0rn/SaltStrap/commit/dd722274c0053aec5491f94c68b320a5a8df2cbd#diff-3af436e6760a0c5f1e3d16a3384853a8

[talwai]

+1. Any updates on this? Will this feature make it into the next release?

[basepi]

It will not, unfortunately. The 2014.7 release is in feature freeze.

This is on our list, and is definitely an oft-requested feature, but we've been focusing on trying to get the bug count down before we focus on feature work.

[cackovic]

+1 The docs should say that gitfs doesn't work when masterless

[plainlystated]

+1 agree with @cackovic ... I just wasted an hour trying to get this to work, because the docs say it does.

[basepi]

@plainlystated Sorry! I created #15444 to solve this doc issue for the time being.

[plainlystated]

@basepi sweet thanks!

[terminalmage]

This has to do with the fact that the LocalClient does not use the salt.fileserver.Fileserver class to allow for the use of the modular fileserver. It hard-codes the use of file_roots.

This means that none of the modular backends work, not just gitfs.

I will look into this.

[howardroark]

+1 This is really the only thing that keeps me from using "formulas". I don't work with large numbers of servers that have a need to stay up for long so I never really end up needing a "master" server. I just use vagrant-aws with Salt in "local" mode and bootstrap all my states with a shell script and git. It would be sooooo much nicer to just configure that bit in the minion as a gitfs.

IMHO this is the key to getting more people using Salt with Vagrant. As of right now you can't really use formulas properly.

[terminalmage]

This should be fixed in 2014.7.0.

[howardroark]

I am super excited if that is true! Looks like that release is about to wrap up as well :)

[howardroark]

@terminalmage Any idea if git_pillar would work under these changes as well?

http://docs.saltstack.com/en/latest/ref/pillar/all/salt.pillar.git_pillar.html#module-salt.pillar.git_pillar

[terminalmage]

@howardroark Regarding the initial content of your last post, the log message was there because Salt has switched to pygit2 as the preferred interface to git in 2014.7.0. GitPython is still supported, and as you can see from the log there, it did detect GitPython.

You can squelch that warning by setting gitfs_provider: gitpython in your minion config.

The git external pillar is not the same thing as gitfs and has not yet been modified work with pygit2. We are targeting a rewrite for one of the next 2 feature releases.

[howardroark]

@terminalmage Thanks for detailed response :) I deleted my last post because I soon realized that my core issue was not related to that message. Great tip though! Seems pygit2 and libgit2 are having install issues on Ubunut 14.04, either way gitpython was a much simpler install.