gitfs backend with root/mountpoint seems doesn't work with Windows

[gigi206]

Description of Issue

I arrive to connect remotly on my git repo with the backend gitfs. But when I try to change root, it seems doesn't work (Windows).

I edit file C:\salt\bin\Lib\site-packages\salt\utils\gitfs.py and I add on line 2348 (version 2019.2) the following code:

from pprint import pprint
pprint(repo.file_list('salt'))

And I have the following output without root:

({'.vscode/settings.json',
  'LICENSE',
  'README.md',
  'salt/xxx/defaults.yaml',
  'salt/xxx/init.sls',
  'top.sls'},
 {})

And with root:

({'xxx\\defaults.yaml',
  'xxx\\init.sls'},
 {})

First has a slash and the second has a backslash. I think that's why I have the issue.

Setup

Note I have the same issue with pygit2 and gitpython. My custom config:

file_client: local
fileserver_backend:
  - roots
  - git
top_file_merging_strategy: same
gitfs_provider: pygit2
# gitfs_provider: gitpython
gitfs_base: salt
gitfs_remotes:
  - https://xxx.git:
    - root: salt
    - mountpoint: salt://
    - base: salt

Steps to Reproduce Issue

Without options root and mountpoint activated return True:

PS C:\WINDOWS\system32> C:\salt\bin\python.exe C:\salt\bin\Scripts\salt-call state.sls_exists salt/xxx
[ERROR   ] Exception caught while fetching gitfs remote 'https://xxx.git': environment can only contain strings
Traceback (most recent call last):
  File "C:\salt\bin\lib\site-packages\salt\utils\gitfs.py", line 2348, in fetch_remotes
    if repo.fetch():
  File "C:\salt\bin\lib\site-packages\salt\utils\gitfs.py", line 767, in fetch
    return self._fetch()
  File "C:\salt\bin\lib\site-packages\salt\utils\gitfs.py", line 1767, in _fetch
    cleaned = self.clean_stale_refs(local_refs=refs_post)
  File "C:\salt\bin\lib\site-packages\salt\utils\gitfs.py", line 1623, in clean_stale_refs
    return super(Pygit2, self).clean_stale_refs()
  File "C:\salt\bin\lib\site-packages\salt\utils\gitfs.py", line 602, in clean_stale_refs
    stderr=subprocess.STDOUT)
  File "C:\salt\bin\lib\subprocess.py", line 676, in __init__
    restore_signals, start_new_session)
  File "C:\salt\bin\lib\subprocess.py", line 957, in _execute_child
    startupinfo)
TypeError: environment can only contain strings
local:
    True

With options root and mountpoint activated return False:

PS C:\WINDOWS\system32> C:\salt\bin\python.exe C:\salt\bin\Scripts\salt-call state.sls_exists xxx
[ERROR   ] Exception caught while fetching gitfs remote 'https://xxx.git': environment can only contain strings
Traceback (most recent call last):
  File "C:\salt\bin\lib\site-packages\salt\utils\gitfs.py", line 2348, in fetch_remotes
    if repo.fetch():
  File "C:\salt\bin\lib\site-packages\salt\utils\gitfs.py", line 767, in fetch
    return self._fetch()
  File "C:\salt\bin\lib\site-packages\salt\utils\gitfs.py", line 1767, in _fetch
    cleaned = self.clean_stale_refs(local_refs=refs_post)
  File "C:\salt\bin\lib\site-packages\salt\utils\gitfs.py", line 1623, in clean_stale_refs
    return super(Pygit2, self).clean_stale_refs()
  File "C:\salt\bin\lib\site-packages\salt\utils\gitfs.py", line 602, in clean_stale_refs
    stderr=subprocess.STDOUT)
  File "C:\salt\bin\lib\subprocess.py", line 676, in __init__
    restore_signals, start_new_session)
  File "C:\salt\bin\lib\subprocess.py", line 957, in _execute_child
    startupinfo)
TypeError: environment can only contain strings
local:
    False

Also note and I don't know why (I haven't debug) that repo.fetch return an error !

Versions Report

PS C:\WINDOWS\system32> C:\salt\bin\python.exe C:\salt\bin\Scripts\salt-call --versions-report
Salt Version:
           Salt: 2019.2.0

Dependency Versions:
           cffi: 1.11.5
       cherrypy: 17.4.1
       dateutil: 2.7.5
      docker-py: Not Installed
          gitdb: 2.0.5
      gitpython: 2.1.10
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: 0.28.2
        libnacl: 1.6.1
       M2Crypto: Not Installed
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.17
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.28.2
         Python: 3.5.4 (v3.5.4:3f56838, Aug  8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)]
   python-gnupg: 0.4.3
         PyYAML: 3.13
          PyZMQ: 17.1.2
           RAET: Not Installed
          smmap: 2.0.5
        timelib: 0.2.4
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist:
         locale: cp1252
        machine: AMD64
        release: 10
         system: Windows
        version: 10 10.0.17134 SP0 Multiprocessor Free

The main issue is the root and mountpoint options and the minor is the traceback TypeError: environment can only contain strings

[twangboy]

@gigi206 Since I don't have access to your repo I've tried to replicate this using the salt-formula repo as described in the gitfs walkthrough: https://docs.saltstack.com/en/latest/topics/tutorials/gitfs.html#simple-configuration

I'm not getting the errors you're seeing. Can you try using the salt-formula repo to see if you get the same errors?

[gigi206]

I take a random formulas => jenkins

With the following settings it works:

file_client: local
fileserver_backend:
  - roots
  - git
top_file_merging_strategy: same
gitfs_provider: pygit2
gitfs_remotes:
  - https://github.com/saltstack-formulas/jenkins-formula.git

PS C:\WINDOWS\system32> C:\salt\bin\python.exe C:\salt\bin\Scripts\salt-call state.sls_exists jenkins/nginx
[ERROR   ] Exception caught while fetching gitfs remote 'https://github.com/saltstack-formulas/jenkins-formula.git': environment can only contain strings
Traceback (most recent call last):
  File "C:\salt\bin\lib\site-packages\salt\utils\gitfs.py", line 2348, in fetch_remotes
    if repo.fetch():
  File "C:\salt\bin\lib\site-packages\salt\utils\gitfs.py", line 767, in fetch
    return self._fetch()
  File "C:\salt\bin\lib\site-packages\salt\utils\gitfs.py", line 1767, in _fetch
    cleaned = self.clean_stale_refs(local_refs=refs_post)
  File "C:\salt\bin\lib\site-packages\salt\utils\gitfs.py", line 1623, in clean_stale_refs
    return super(Pygit2, self).clean_stale_refs()
  File "C:\salt\bin\lib\site-packages\salt\utils\gitfs.py", line 602, in clean_stale_refs
    stderr=subprocess.STDOUT)
  File "C:\salt\bin\lib\subprocess.py", line 676, in __init__
    restore_signals, start_new_session)
  File "C:\salt\bin\lib\subprocess.py", line 957, in _execute_child
    startupinfo)
TypeError: environment can only contain strings
local:
    True

And with these settings it doesn't work:

file_client: local
fileserver_backend:
  - roots
  - git
top_file_merging_strategy: same
gitfs_provider: pygit2
gitfs_remotes:
  - https://github.com/saltstack-formulas/jenkins-formula.git:
    - root: jenkins
    - mountpoint: salt://

PS C:\WINDOWS\system32> C:\salt\bin\python.exe C:\salt\bin\Scripts\salt-call state.sls_exists nginx
[ERROR   ] Exception caught while fetching gitfs remote 'https://github.com/saltstack-formulas/jenkins-formula.git': environment can only contain strings
Traceback (most recent call last):
  File "C:\salt\bin\lib\site-packages\salt\utils\gitfs.py", line 2350, in fetch_remotes
    if repo.fetch():
  File "C:\salt\bin\lib\site-packages\salt\utils\gitfs.py", line 767, in fetch
    return self._fetch()
  File "C:\salt\bin\lib\site-packages\salt\utils\gitfs.py", line 1767, in _fetch
    cleaned = self.clean_stale_refs(local_refs=refs_post)
  File "C:\salt\bin\lib\site-packages\salt\utils\gitfs.py", line 1623, in clean_stale_refs
    return super(Pygit2, self).clean_stale_refs()
  File "C:\salt\bin\lib\site-packages\salt\utils\gitfs.py", line 602, in clean_stale_refs
    stderr=subprocess.STDOUT)
  File "C:\salt\bin\lib\subprocess.py", line 676, in __init__
    restore_signals, start_new_session)
  File "C:\salt\bin\lib\subprocess.py", line 957, in _execute_child
    startupinfo)
TypeError: environment can only contain strings
local:
    False

The following code that I add (explain in 1st send):

pprint(repo.file_list('base'))

return with the 1st config:

({'.gitignore',
  '.travis.yml',
  'Gemfile',
  'LICENSE',
  'README.rst',
  'jenkins/cli.sls',
  'jenkins/defaults.yaml',
  'jenkins/files/Debian/jenkins.conf',
  'jenkins/files/RedHat/jenkins.conf',
  'jenkins/files/nginx.conf',
  'jenkins/init.sls',
  'jenkins/jobs.sls',
  'jenkins/macros.jinja',
  'jenkins/map.jinja',
  'jenkins/nginx.sls',
  'jenkins/osmap.yaml',
  'jenkins/plugins.sls',
  'kitchen.yml',
  'pillar.example',
  'test/integration/default/jenkins_spec.rb',
  'test/pillar/example.sls',
  'test/pillar/nginx.sls',
  'test/pillar/top.sls'},
 {'test/pillar/example.sls': b'../../pillar.example'})

And with the 2nd:

({'cli.sls',
  'defaults.yaml',
  'files\\Debian\\jenkins.conf',
  'files\\RedHat\\jenkins.conf',
  'files\\nginx.conf',
  'init.sls',
  'jobs.sls',
  'macros.jinja',
  'map.jinja',
  'nginx.sls',
  'osmap.yaml',
  'plugins.sls'},
 {})

[twangboy]

working minion config:

file_client: local
fileserver_backend:
  - roots
  - git
gitfs_provider: gitpython
gitfs_base: master
gitfs_remotes:
  - https://github.com/saltstack-formulas/jenkins-formula.git

Results

C:\dev\salt>salt-call cp.list_master
local:
    - .gitignore
    - .travis.yml
    - Gemfile
    - LICENSE
    - README.rst
    - jenkins/cli.sls
    - jenkins/defaults.yaml
    - jenkins/files/Debian/jenkins.conf
    - jenkins/files/RedHat/jenkins.conf
    - jenkins/files/nginx.conf
    - jenkins/init.sls
    - jenkins/jobs.sls
    - jenkins/macros.jinja
    - jenkins/map.jinja
    - jenkins/nginx.sls
    - jenkins/osmap.yaml
    - jenkins/plugins.sls
    - kitchen.yml
    - pillar.example
    - test/integration/default/jenkins_spec.rb
    - test/pillar/example.sls
    - test/pillar/nginx.sls
    - test/pillar/top.sls

C:\dev\salt>salt-call state.sls_exists jenkins/nginx
local:
    True

Adding root and mountpoint in the minion config like so:

file_client: local
fileserver_backend:
  - roots
  - git
gitfs_provider: gitpython
gitfs_base: master
gitfs_remotes:
  - https://github.com/saltstack-formulas/jenkins-formula.git:
    - root: jenkins
    - mountpoint: salt://formulas

Results:

C:\dev\salt>salt-call cp.list_master
local:
    - formulas/cli.sls
    - formulas/defaults.yaml
    - formulas/files\Debian\jenkins.conf
    - formulas/files\RedHat\jenkins.conf
    - formulas/files\nginx.conf
    - formulas/init.sls
    - formulas/jobs.sls
    - formulas/macros.jinja
    - formulas/map.jinja
    - formulas/nginx.sls
    - formulas/osmap.yaml
    - formulas/plugins.sls

C:\dev\salt>salt-call state.sls_exists formulas/nginx
local:
    False

C:\dev\salt>salt-call state.sls_exists formulas\\nginx
local:
    False

C:\dev\salt>salt-call state.sls_exists formulas\nginx
local:
    False

C:\dev\salt>salt-call state.sls_exists "formulas\nginx"
local:
    False

C:\dev\salt>salt-call state.sls_exists "formulas\\nginx"
local:
    False

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[amalaguti]

Hi, just to mention this is still an issue on recent versions.

[amalaguti]

@twangboy any thoughts on this one ?

[amalaguti]

Worth to mention that the minion (running as a service already) looks like it does not seem to be trying to fetch files till you execute some command like the one below

PS C:\ProgramData\Salt Project\Salt\var\cache\salt\minion> salt-call cp.list_master
[INFO    ] Wrote new gitfs remote map to C:\ProgramData\Salt Project\Salt\var\cache\salt\minion\gitfs\remote_map.txt

Till then the minion cache folder does not have a gitfs folder present.