pygit2 doesn't work with salt ssh

edgan commented 8 years ago

Description of Issue/Question

I recently switched from filesystem to pygit2/gitfs on my salt master. I am using salt-ssh for development, and found another bug where they didn't behave the same. I ended up tracking it down to filesystem vs gitfs. Then I decided to run salt-ssh with pygit2/gitfs, and found this bug.

Setup

_ssh/Saltfile:
salt-ssh:
  config_dir: .
  max_procs: 1
  ssh_wipe: True

_ssh/master:
cachedir: /var/tmp/salt-ssh/cache

ssh_user: user 
ssh_priv: /home/user/.ssh/id_rsa

state_output: mixed

ssh_minion_opts:
  failhard: True

log_file: /dev/null

gitfs_provider: pygit2

fileserver_backend:
  - git

ext_pillar:
  - git:
    - master git@github.com:foo/salt-pillars:
      - pubkey: /home/user/.ssh/id_rsa.pub
      - privkey: /home/user/.ssh/id_rsa

gitfs_remotes:
  - git@github.com:foo/salt-formulas.git:
    - pubkey: /home/user/.ssh/id_rsa.pub
    - privkey: /home/user/.ssh/id_rsa

Steps to Reproduce Issue

Command:

salt-ssh 'salt-salt-master-01.inf.us-west-2.aws.plume.tech' state.highstate

Log:

[ERROR   ] Template was specified incorrectly: False
[ERROR   ] Template was specified incorrectly: False
[ERROR   ] Template was specified incorrectly: False
[ERROR   ] Template was specified incorrectly: False
[ERROR   ] Template was specified incorrectly: False
[ERROR   ] Template was specified incorrectly: False
[ERROR   ] Template was specified incorrectly: False
[ERROR   ] Template was specified incorrectly: False
[ERROR   ] Template was specified incorrectly: False
[ERROR   ] Template was specified incorrectly: False
[ERROR   ] Template was specified incorrectly: False
[ERROR   ] Template was specified incorrectly: False
[ERROR   ] Template was specified incorrectly: False
[ERROR   ] Template was specified incorrectly: False
[ERROR   ] Template was specified incorrectly: False
[ERROR   ] Template was specified incorrectly: False
[ERROR   ] Template was specified incorrectly: False
[ERROR   ] Template was specified incorrectly: False
[ERROR   ] Template was specified incorrectly: False
[ERROR   ] Template was specified incorrectly: False
[ERROR   ] Template was specified incorrectly: False
[ERROR   ] Template was specified incorrectly: False
[ERROR   ] Template was specified incorrectly: False
[ERROR   ] Template was specified incorrectly: False
[ERROR   ] Template was specified incorrectly: False
[ERROR   ] Template was specified incorrectly: False
[ERROR   ] Template was specified incorrectly: False
[ERROR   ] Template was specified incorrectly: False
[ERROR   ] Template was specified incorrectly: False
[ERROR   ] Template was specified incorrectly: False
salt-salt-master-01.inf.us-west-2.aws.plume.tech:
    - Specified SLS hosts in saltenv base is not available on the salt master or through a configured fileserver
    - Specified SLS resolv_conf in saltenv base is not available on the salt master or through a configured fileserver
    - Specified SLS timezone in saltenv base is not available on the salt master or through a configured fileserver
    - Specified SLS apt in saltenv base is not available on the salt master or through a configured fileserver
    - Specified SLS ubuntu in saltenv base is not available on the salt master or through a configured fileserver
    - Specified SLS bash in saltenv base is not available on the salt master or through a configured fileserver
    - Specified SLS zsh in saltenv base is not available on the salt master or through a configured fileserver
    - Specified SLS root in saltenv base is not available on the salt master or through a configured fileserver
    - Specified SLS profile in saltenv base is not available on the salt master or through a configured fileserver
    - Specified SLS accounts in saltenv base is not available on the salt master or through a configured fileserver
    - Specified SLS base.pkgs in saltenv base is not available on the salt master or through a configured fileserver
    - Specified SLS bind.utils in saltenv base is not available on the salt master or through a configured fileserver
    - Specified SLS cloud_init in saltenv base is not available on the salt master or through a configured fileserver
    - Specified SLS git in saltenv base is not available on the salt master or through a configured fileserver
    - Specified SLS nagios.plugins in saltenv base is not available on the salt master or through a configured fileserver
    - Specified SLS python in saltenv base is not available on the salt master or through a configured fileserver
    - Specified SLS vim in saltenv base is not available on the salt master or through a configured fileserver
    - Specified SLS scripts in saltenv base is not available on the salt master or through a configured fileserver
    - Specified SLS volume in saltenv base is not available on the salt master or through a configured fileserver
    - Specified SLS ec2_local_storage in saltenv base is not available on the salt master or through a configured fileserver
    - Specified SLS sensu.client in saltenv base is not available on the salt master or through a configured fileserver
    - Specified SLS ssh.client in saltenv base is not available on the salt master or through a configured fileserver
    - Specified SLS ssh.server in saltenv base is not available on the salt master or through a configured fileserver
    - Specified SLS datadog in saltenv base is not available on the salt master or through a configured fileserver
    - Specified SLS saltstack.minion in saltenv base is not available on the salt master or through a configured fileserver
    - No matching sls found for 'deployer' in env 'base'
    - No matching sls found for 'foreman' in env 'base'
    - No matching sls found for 'saltstack.api' in env 'base'
    - No matching sls found for 'saltstack.master' in env 'base'
    - No matching sls found for 'saltstack.highstate' in env 'base'

Versions Report

Salt Version:
           Salt: 2016.3.1

Dependency Versions:
           cffi: 1.5.2
       cherrypy: Not Installed
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: 0.24.0
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.7
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
         pygit2: 0.24.1
         Python: 2.7.11 (default, Jun 21 2016, 09:15:12)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.7.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.3
            ZMQ: 4.1.4

System Versions:
           dist: fedora 24 Twenty Four
        machine: x86_64
        release: 4.6.3-300.fc24.x86_64
         system: Linux
        version: Fedora 24 Twenty Four

Ch3LL commented 8 years ago

@edgan I am having a hard time trying to replicate this on 2016.3.1-centos7. Here is my setup. I tried ot keep it exactly as your configs but for my repos:

I've created a _ssh dir: /srv/salt/_ssh
In that directory i have added 3 files: master, roster and Saltfile
Here is my master:

cachedir: /var/tmp/salt-ssh/cache

ssh_user: user 
ssh_priv: /home/user/.ssh/id_rsa

state_output: mixed

ssh_minion_opts:
  failhard: True

log_file: /dev/null

gitfs_provider: pygit2

fileserver_backend:
  - git

ext_pillar:
  - git:
    - master git@github.com:Ch3LL/Ch3LLScripts.git:
      - pubkey: /home/ch3ll/.ssh/int.pub
      - privkey: /home/ch3ll/.ssh/int

gitfs_remotes:
  - git@github.com:Ch3LL/Ch3LLScripts.git:
    - root: salt/states
    - pubkey: /home/ch3ll/.ssh/int.pub
    - privkey: /home/ch3ll/.ssh/int
top_file_merging_strategy: same

Here is my Saltfile

salt-ssh:
  config_dir: .
  max_procs: 1
  ssh_wipe: True

Here is my roster file

ch3ll-test: {host: <ipaddress>}

Then when I run sudo salt-ssh ch3ll-test state.highstate -ldebug the highstate runs correctly and I do not see any errors. Is there anything about my test case that you can point out that I'm doing wrong?

The-Loeki commented 7 years ago

I'm actually using it like this on 2016.11 and it works as advertised (well, it doesn't but that's a different story/issue ;))

Ch3LL commented 7 years ago

Thanks for the input @The-Loeki

@edgan are you still running into this issue?

edgan commented 7 years ago

Yes, it still doesn't work for me. I think it is my use of "include:" in the top.sls.

If I don't set top_file_merging_strategy in the master file I get the errors below. Which just seems to be the same problem with less output.

Versions:

salt-ssh-2016.11.3 python2-pygit2-0.24.2

Errors:

[ngrennan@higgs _ssh_master_dev](master|✔) salt-ssh 'salt-salt-master-01.inf.us-west-2.aws.plume.tech' state.highstate
[ERROR   ] Rendering exception occurred: Jinja variable 'dict object' has no attribute 'is_bento'
[CRITICAL] Rendering SLS 'base:base' failed: Jinja variable 'dict object' has no attribute 'is_bento'
[ERROR   ] Template was specified incorrectly: False
salt-salt-master-01.inf.us-west-2.aws.plume.tech:
    - Rendering SLS 'base:base' failed: Jinja variable 'dict object' has no attribute 'is_bento'
    - No matching sls found for 'deployer' in env 'base'

If I use "top_file_merging_strategy: same" in the master file, I get the exact same errors as before.

top.sls:

base:
  '* and not G@is_ssh:2':
    - base

  '* and not G@ops:env:inf and not G@is_ssh:2':
    - deployer

  'G@ops:component_cluster:web-neptune and not G@is_ssh:2':
    - neptune

include:
  - data
  - deployments
  - deploys
  - infrastructure
  - production
  - shared

edgan commented 7 years ago

These errors suggest that it is not picking up my custom python grains from _grains when using gitfs. This does work when using the filesystem method.

Errors:

[ERROR   ] Rendering exception occurred: Jinja variable 'dict object' has no attribute 'is_bento'
[CRITICAL] Rendering SLS 'base:base' failed: Jinja variable 'dict object' has no attribute 'is_bento'

stale[bot] commented 6 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

saltstack / salt