windows minion misses pygit2 or GitPython lib

[TheBigBear]

On a 2015.8.0rc3 windows minion the pygit2 or GitPython libs are not installed by default.

That makes the masterless minions work with git_remotes in the new intended way a bit challenging, I think.

see notes about changes in 2015.8 on http://docs.saltstack.com/en/develop/topics/windows/windows-package-manager.html

it says:

Starting in version 2015.8.0, the winrepo.update_git_repos runner now makes use of the same underlying code used by the Git Fileserver Backend and Git External Pillar to maintain and update its local clones of git repositories. If a compatible version of either pygit2 (0.20.3 and later) or GitPython (0.3.0 or later) is installed, then Salt will use it instead of the old method (which invokes the git.latest state).

Note

If compatible versions of both pygit2 and GitPython are installed, then Salt will prefer pygit2, to override this behavior use the winrepo_provider configuration parameter:

winrepo_provider: gitpython

I believe it would be a good idea to add either or both of pygit2 and/or Git Python.

[jfindlay]

@TheBigBear, thanks for the report.

[terminalmage]

At the very least, we should be falling back to the old code if we can't provide gitpython/pygit2. @twangboy please work with me on this.

[terminalmage]

@TheBigBear winrepo should still work the same as before on masterless, the gitpython/pygit2 stuff has only been done for the winrepo runner so far. Masterless uses the winrepo execution module, which is completely different code.

[twangboy]

@terminalmage Dave mentioned to me that we should take the work you've done on the runner and implement it for masterless. He said they should pretty much work the same.

[terminalmage]

@twangboy Way ahead of you, already did it this afternoon :smile:

Just fleshing out the masterless winrepo documentation right now.

[terminalmage]

We can't quite use pygit2/gitpython in the winrepo execution module, but it will use git.latest now: https://github.com/saltstack/salt/pull/26541

[terminalmage]

With that pull request, it is now more explicit that masterless does not support the new pygit2/gitpython support. Closing this issue.

[TheBigBear]

@terminalmage can I ask why? I have both pygit2 and GitPython working as python libs in my 2015.8.0rc03+ ? What is stopping it from pygit2 and GitPython libs to be added to the windows salt-minion installer?

[TheBigBear]

@terminalmage when is your masterless-winrepo branch going to end up in 2015.8 , do you know?

[terminalmage]

@TheBigBear Nothing is stopping it, we just don't provide yet in our installer, and I haven't tested it.

The code does check now to see if a compatible version of pygit2/gitpython is available, and tries to use it if it's there. So, it technically should be supported, provided that gitpython or pygit2 is available to the masterless minion. But, since we provide a self-contained python distribution with Salt, and it's not present there (yet), I can't guarantee it'll work. If you have these modules added to your system, by all means, please test it and let me know if it works.

My pull request should be merged sometime on Friday, pending internal code review by our QA staff and the results of our unit tests.

[TheBigBear]

@terminalmage thanks for all your explanations. What I wanted was to ask saltstack to please put pygit2 and/or GitPython in the stock saltstack windows minion installer. I have spent weeks now updating the winrepo adding the new jinja templating all under 2015.8 and would love to have 2015.8 come out with it working on masterless as well as with a master. So my aim of this particular PR was always specifically to do with >= 2015.8 Berryllium and up. Haivng it closed make me wonder how do I get the pygit2 and GitPython libs to be included?

[terminalmage]

@TheBigBear it would have to be included in our installer. We don't yet have it there, and may not have time to include it in time for the release, but don't worry; this will not stop you from using the winrepo in 2015.8.0. The winrepo will fall back to using the git.latest state to clone/update any repos configured in winrepo_remotes. So, you do have something to fall back on here.

Note that git.latest requires Git for Windows to be installed, and when installing you must select one of the install options that makes the git command executable from the Command Prompt.

[TheBigBear]

@terminalmage ok, got that, thanks. The remaining issue I have with this is that the current 'git.latest' presumably creates a c:\salt\srv\salt\win\repo\salt-winrepo.git directory, right? (just an assumption?) and if it does it makes the sls files inaccessible to state.show_sls, because the path to them contains a ".".

But if it used pygit2 the salt-winrepo.git directory name would actually be some hashname instead without a "." in it and it would be possible to use state.show_sls on it's files. (just assumptions on my part - which is why I closed the #26158 issue myself, counting on pygit2 to be coming out on the 2015.8 rel.)

see #26158 and https://github.com/saltstack/salt/pull/26422

[TheBigBear]

@UtahDave ^^^ @twangboy who can I best work with to try to get pygit2 (and GitPython) libs added the the python stock libs in the rc4 or imminent 2015.8 windows salt-minion installer?

I am really hoping this can still make it in somehow?

@UtahDave how are the ducks lining up on the windows side? Have you had your meeting with Shane as yet? If you are having it today, I might be able to join, as I have no major 'blocks' of time dominated by agenda items today. I haven't seen any word on the new window repo ...?

[terminalmage]

@TheBigBear I have pushed an update to the code that handles checking out the git repo using git.latest. Before, the way that it came up with the directory names was to just use the last bit of the URL. So https://github.com/saltstack/salt-winrepo.git would result in the files being cloned into a directory called salt-winrepo.git. With the commit that I just added to this pull request, this would now become salt-winrepo_git.

[TheBigBear]

@terminalmage wow that was quick! Awesome. You don't need a lot of sleep? ;-)

Yeah, I guess that way we have a bit more time to test pygit2 (or GitPython) but still don't shoot ourselves in the foot with the "." in the directory name path. Well done.

Just on the off-chance that you do know the answer to this, is jfindlay or twangboy or even UtahDave the best person to talk to about getting the pygit2 and GitPython libs into the std windows salt minion installer?

I tried raising another issue for it on the salt-windows-dev repo https://github.com/saltstack/salt-windows-dev/issues/23 so the closing of this one https://github.com/saltstack/salt/issues/26461 won't matter.

[TheBigBear]

@terminalmage never mind I found the commit 38136b37572029ce81e1fb570f932ee1eaad2137 and it is in the PR. Awesome.

[terminalmage]

Yep that's the one!