search

ytti / oxidized

Oxidized is a network device configuration backup tool. It's a RANCID replacement!
Apache License 2.0
2.79k stars 922 forks source link

Hook Remote git error "wrong argument type nil " #497

Closed totosh closed 8 years ago

totosh commented 8 years ago

I'm trying to setup remote repo git but i get this error :

E, [2016-07-13T11:40:43.201248 #12005] ERROR -- : Hook push_to_remote_repo (#<GithubRepo:0x0000000221cda0>) failed (#<TypeError: wrong argument type nil (expected String)>) for event :post_store
D, [2016-07-13T11:40:43.201307 #12005] DEBUG -- : lib/oxidized/worker.rb: Jobs 0, Want: 1

$ oxidized -v 0.15.0

I get this error with 2.1.2 and 2.1.8 Ruby version.

Below my config :

pid:
"/var/run/oxidized/oxidized.pid"
log: /var/log/oxidized/log
use_syslog: false
username: #####
password: #####
model: cisco
interval: 10800
debug: true
threads: 30
timeout: 20
retries: 3
prompt: !ruby/regexp /^([\w.@-]+[#>]\s?)$/
rest: #####
vars: {}
groups: 
  net-cisco-backup:
    username: #####
    password: #####
input:
  default: ssh
  debug: false
  ssh:
    secure: false
output:
  default: git
  git: 
    user: #####
    email: #####
    repo: "/home/oxidized/git-repos/default.git" 
source:
  default: sql
  sql:
    adapter: sqlite
    database: "/etc/oxidized/oxidized.db"
    table: devices
    map:
      name: address
      model: model
      group: groupname
      username: username
      password: password
      enable: enable
model_map:
  cisco: ios
  juniper: junos
hooks:
  push_to_remote_repo:
    type: githubrepo
    remote_repo:
      net-cisco-backup: "git@gitlab.domain.lan:10022/oxidized/net-cisco-backup.git"
    events: [post_store]
    publickey: /home/oxidized/.ssh/id_rsa.pub
    privatekey: /home/oxidized/.ssh/id_rsa

Local repo work fine.

Totosh.

ytti commented 8 years ago

Do you have crash file from ~/.config/oxidized I'd like to see the traceback.

totosh commented 8 years ago

There is no crash file with this error. I only see this error in the log file.

ytti commented 8 years ago

Is your node group net-cisco-backup? If not, it won't work. It must match to your group.

If you don't have group, or you don't plan to put different groups to different repos. Then just change it to remote_repo: "git@gitlab.domain.lan:10022/oxidized/net-cisco-backup.git"

totosh commented 8 years ago

Yes, My node group is net-cisco-backup.

Here my DB sqlite :

sqlite> .tables
devices
sqlite> .schema devices
CREATE TABLE devices (
id INTEGER PRIMARY KEY AUTOINCREMENT,
hostname,
address,
model,
groupname,
username,
password,
enable,
prompt,
proxy,
description
);
sqlite> select * from devices;
1|RSWBB-DCB-01|10.#######|cisco|net-cisco-backup||||||

My repo local is well created

git-repos/
└── net-cisco-backup.git
    ├── config
    ├── description
    ├── HEAD
    ├── hooks
    │   └── README.sample
    ├── index
    ├── info
    │   └── exclude
    ├── objects
    │   ├── 4b
    │   │   └── 825dc642cb6eb9a060e54bf8d69288fbee4904
    │   ├── 7d
    │   │   └── 5048f78468e8ccba40283049231686a4caaba1
    │   ├── a2
    │   │   └── 901766833e2de213bdf867a67cf75527272084
    │   ├── f8
    │   │   └── 7a7696662d6609b3f2cdd7b20b5ed1da21422c
    │   ├── info
    │   └── pack
    └── refs
        ├── heads
        │   └── master
        └── tags
totosh commented 8 years ago

I tried to switch into a single group config. I added single_repo: true and remote_repo: "git@gitlab.domain.lan:10022/oxidized/net-cisco-backup.git", that works.

So the problem concern groups repositories.

ytti commented 8 years ago

Can you try if the '-' is changed to '_' on the group name? Or install pry: gem install pry then in https://github.com/ytti/oxidized/blob/master/lib/oxidized/hook/githubrepo.rb add require 'pry'then after line 65 add binding.pry.

Now when you run oxidized, it should show you REPL prompt once hitting that piece of code. On that prompt type p cfg.remote_repoand p node.group output of those should give us more insight into the issue.

totosh commented 8 years ago

I rename net-cisco-backup by "net_cisco_backup" then "cisco", i get the same error.

Before error log, i saw this line :

D, [2016-07-13T17:41:56.047466 #4978] DEBUG -- : Hook "push_to_remote_repo" registered GithubRepo for event :post_store

In the file git-repos/net-cisco-backup.git/config, remote config is not present.

[core]
    bare = true
    filemode = true
[core]
    repositoryformatversion = 0
[user]
    name = ##############
[user]
    email = #####@######

I added in /usr/local/rvm/gems/ruby-2.1.8/gems/oxidized-0.15.0/lib/oxidized/hook/githubrepo.rb require 'pry' after line 1

class GithubRepo < Oxidized::Hook
  require 'pry'
  def validate_cfg!
  ...

and binding.pry after line 65

  def remote_repo(node)
    if node.group.nil? || cfg.remote_repo.is_a?(String)
      cfg.remote_repo
    else
      binding.pry
      cfg.remote_repo[node.group]
    end
  end

I don't see anything :s

[oxidized@vprsaltmaster74 ~]$ oxidized  -d
Puma 2.16.0 starting...
* Min threads: 0, max threads: 16
* Environment: development
* Listening on tcp://10.100.20.74:8888
totosh commented 8 years ago

I'm not ruby developper but i move binding.pry at line 8

  7   def run_hook(ctx)
  8     binding.pry
  9     repo = Rugged::Repository.new(ctx.node.repo)

And i do that :

From: /usr/local/rvm/gems/ruby-2.1.8/gems/oxidized-0.15.0/lib/oxidized/hook/githubrepo.rb @ line 8 GithubRepo#run_hook:

     7: def run_hook(ctx)
 =>  8:   binding.pry
     9:   repo = Rugged::Repository.new(ctx.node.repo)
    10:   log "Pushing local repository(#{repo.path})..."
    11:   remote = repo.remotes['origin'] || repo.remotes.create('origin', remote_repo(ctx.node))
    12:   log "to remote: #{remote.url}"
    13: 
    14:   fetch_and_merge_remote(repo)
    15: 
    16:   remote.push([repo.head.name], credentials: credentials)
    17: end

[1] pry(#<GithubRepo>)> repo = Rugged::Repository.new(ctx.node.repo)
TypeError: wrong argument type nil (expected String)
from (pry):1:in `new'
[2] pry(#<GithubRepo>)>
ytti commented 8 years ago

Ok it failed earlier than I thought.

Can you type in the same prompt:

furhouse commented 8 years ago

I've ran into the same issue, after adding require 'pry' & binding.pry (also on line 8, like totosh), I get this output:

root@debian-jessie:~# oxidized -d
I, [2016-07-14T08:45:56.493243 #2711]  INFO -- : Oxidized starting, running as pid 2711
D, [2016-07-14T08:45:56.612703 #2711] DEBUG -- : Hook "push_to_remote" registered GithubRepo for event :node_success
resolving DNS for testsrx.location.example.local...
resolving DNS for testubnt.location.example.local...
resolving DNS for testprocurve.location.example.local...
I, [2016-07-14T08:45:56.683851 #2711]  INFO -- : Loaded 3 nodes
D, [2016-07-14T08:45:57.014784 #2711] DEBUG -- : Jobs 0, Want: 1
Puma 2.16.0 starting...
* Min threads: 0, max threads: 16
* Environment: development
* Listening on tcp://0.0.0.0:80
D, [2016-07-14T08:45:58.063105 #2711] DEBUG -- : SSH: show configuration | display omit @ testsrx.location.example.local
D, [2016-07-14T08:45:59.663826 #2711] DEBUG -- : SSH: show version @ testsrx.location.example.local
D, [2016-07-14T08:46:00.423550 #2711] DEBUG -- : SSH: show chassis hardware @ testsrx.location.example.local

From: /var/lib/gems/2.1.0/gems/oxidized-0.14.3/lib/oxidized/hook/githubrepo.rb @ line 8 GithubRepo#run_hook:

     7: def run_hook(ctx)
 =>  8:   binding.pry
     9:   repo = Rugged::Repository.new(ctx.node.repo)
    10:   log "Pushing local repository(#{repo.path})..."
    11:   remote = repo.remotes['origin'] || repo.remotes.create('origin', remote_repo(ctx.node))
    12:   log "to remote: #{remote.url}"
    13:
    14:   fetch_and_merge_remote(repo)
    15:
    16:   remote.push([repo.head.name], credentials: credentials)
    17: end

[1] pry(#<GithubRepo>)> p ctx
#<Oxidized::HookManager::HookContext node=#<Oxidized::Node:0x000000020cdff8 @name="testsrx.location.example.local", @ip="10.53.14.230", @group="ens", @input=[Oxidized::SSH], @output=Oxidized::Git, @model=#<JunOS:0x00000001fa34c0 @node=#<Oxidized::Node:0x000000020cdff8 ...>, @input=nil, @model="srx240h2">, @auth={:username=>"oxidized", :password=>"TxYSfgt5HgM6cH66R0s00vYY"}, @prompt=/^([\w.@-]+[#>]\s?)$/, @vars={:comware_cmdline=>nil}, @stats=#<Oxidized::Node::Stats:0x00000001fa23e0 @stats={:success=>[{:start=>2016-07-14 08:45:57 UTC, :end=>2016-07-14 08:46:01 UTC, :time=>4.329201048}]}>, @retry=0, @repo=nil, @running=false, @last=#<OpenStruct start=2016-07-14 08:45:57 UTC, end=2016-07-14 08:46:01 UTC, status=:success, time=4.329201048>>, job=#<Oxidized::Job:0x00000002cd97e8 dead>, event=:node_success>
=> #<Oxidized::HookManager::HookContext node=#<Oxidized::Node:0x000000020cdff8 @name="testsrx.location.example.local", @ip="10.53.14.230", @group="ens", @input=[Oxidized::SSH], @output=Oxidized::Git, @model=#<JunOS:0x00000001fa34c0 @node=#<Oxidized::Node:0x000000020cdff8 ...>, @input=nil, @model="srx240h2">, @auth={:username=>"oxidized", :password=>"TxYSfgt5HgM6cH66R0s00vYY"}, @prompt=/^([\w.@-]+[#>]\s?)$/, @vars={:comware_cmdline=>nil}, @stats=#<Oxidized::Node::Stats:0x00000001fa23e0 @stats={:success=>[{:start=>2016-07-14 08:45:57 UTC, :end=>2016-07-14 08:46:01 UTC, :time=>4.329201048}]}>, @retry=0, @repo=nil, @running=false, @last=#<OpenStruct start=2016-07-14 08:45:57 UTC, end=2016-07-14 08:46:01 UTC, status=:success, time=4.329201048>>, job=#<Oxidized::Job:0x00000002cd97e8 dead>, event=:node_success>
[2] pry(#<GithubRepo>)> p ctx.node
#<Oxidized::Node:0x000000020cdff8 @name="testsrx.location.example.local", @ip="10.53.14.230", @group="ens", @input=[Oxidized::SSH], @output=Oxidized::Git, @model=#<JunOS:0x00000001fa34c0 @node=#<Oxidized::Node:0x000000020cdff8 ...>, @input=nil, @model="srx240h2">, @auth={:username=>"oxidized", :password=>"TxYSfgt5HgM6cH66R0s00vYY"}, @prompt=/^([\w.@-]+[#>]\s?)$/, @vars={:comware_cmdline=>nil}, @stats=#<Oxidized::Node::Stats:0x00000001fa23e0 @stats={:success=>[{:start=>2016-07-14 08:45:57 UTC, :end=>2016-07-14 08:46:01 UTC, :time=>4.329201048}]}>, @retry=0, @repo=nil, @running=false, @last=#<OpenStruct start=2016-07-14 08:45:57 UTC, end=2016-07-14 08:46:01 UTC, status=:success, time=4.329201048>>
=> #<Oxidized::Node:0x000000020cdff8
 @auth={:username=>"oxidized", :password=>"TxYSfgt5HgM6cH66R0s00vYY"},
 @group="ens",
 @input=[Oxidized::SSH],
 @ip="10.53.14.230",
 @last=#<OpenStruct start=2016-07-14 08:45:57 UTC, end=2016-07-14 08:46:01 UTC, status=:success, time=4.329201048>,
 @model=#<JunOS:0x00000001fa34c0 @input=nil, @model="srx240h2", @node=#<Oxidized::Node:0x000000020cdff8 ...>>,
 @name="testsrx.location.example.local",
 @output=Oxidized::Git,
 @prompt=/^([\w.@-]+[#>]\s?)$/,
 @repo=nil,
 @retry=0,
 @running=false,
 @stats=#<Oxidized::Node::Stats:0x00000001fa23e0 @stats={:success=>[{:start=>2016-07-14 08:45:57 UTC, :end=>2016-07-14 08:46:01 UTC, :time=>4.329201048}]}>,
 @vars={:comware_cmdline=>nil}>
[3] pry(#<GithubRepo>)> p ctx.node.repo
nil
=> nil
[4] pry(#<GithubRepo>)>

The relevant hook:

hooks:
  push_to_remote:
    type: githubrepo
    remote_repo:
      ens: git@git.example.com:oxidized/ens.git
      lab: git@git.example.com:oxidized/lab.git
    events: [node_success]
    publickey: /root/.ssh/id_rsa.pub
    privatekey: /root/.ssh/id_rsa
ytti commented 8 years ago

So it seems this will result in nil: https://github.com/ytti/oxidized/blob/master/lib/oxidized/node.rb#L25

Which runs this: https://github.com/ytti/oxidized/blob/master/lib/oxidized/node.rb#L174

I struggle to understand how it could possibly be nil.

Could you add in the pry output:

Also could you change https://github.com/ytti/oxidized/blob/master/lib/oxidized/node.rb#L177 String to ::String.

totosh commented 8 years ago

The output :

[oxidized@vprsaltmaster74 ~]$ oxidized
Puma 2.16.0 starting...
* Min threads: 0, max threads: 16
* Environment: development
* Listening on tcp://10.100.20.74:8888

From: /usr/local/rvm/gems/ruby-2.1.8/gems/oxidized-0.15.0/lib/oxidized/hook/githubrepo.rb @ line 8 GithubRepo#run_hook:

     7: def run_hook(ctx)
 =>  8:   binding.pry
     9:   repo = Rugged::Repository.new(ctx.node.repo)
    10:   log "Pushing local repository(#{repo.path})..."
    11:   remote = repo.remotes['origin'] || repo.remotes.create('origin', remote_repo(ctx.node))
    12:   log "to remote: #{remote.url}"
    13:
    14:   fetch_and_merge_remote(repo)
    15:
    16:   remote.push([repo.head.name], credentials: credentials)
    17: end

[3] pry(#<GithubRepo>)> p ctx.node.repo
nil
=> nil
[4] pry(#<GithubRepo>)> p Oxidized.config.output.git.repo
"/etc/oxidized/git-repos/default.git"
=> "/etc/oxidized/git-repos/default.git"
[5] pry(#<GithubRepo>)> p Oxidized.config.output.git.repo.is_a?(String)
true
=> true
[6] pry(#<GithubRepo>)> p Oxidized.config.output.git.single_repo
#<Asetus::ConfigStruct:0x0000000480a1c0 @key_to_s=nil, @cfg={}>
=> #<Asetus::ConfigStruct:0x0000000480a1c0 @cfg={}, @key_to_s=nil>
[7] pry(#<GithubRepo>)>

After changing String to ::String in https://github.com/ytti/oxidized/blob/master/lib/oxidized/node.rb#L177

New output :

[1] pry(#<GithubRepo>)> p ctx
#<Oxidized::HookManager::HookContext node=#<Oxidized::Node:0x0000000392a0b0 @name="10.100.3.254", @ip="10.100.3.254", @group="net-cisco-backup", @input=[Oxidized::SSH], @output=Oxidized::Git, @model=#<IOS:0x000000036e6560 @node=#<Oxidized::Node:0x0000000392a0b0 ...>, @input=nil>, @auth={:username=>"bauxia", :password=>"Sd2D@D83LceP@33mu-2"}, @prompt=/^([\w.@()-]+[#>]\s?)$/, @vars=nil, @stats=#<Oxidized::Node::Stats:0x000000036e4f30 @stats={:success=>[{:start=>2016-07-14 09:20:18 UTC, :end=>2016-07-14 09:20:24 UTC, :time=>5.879200668}]}>, @retry=0, @repo="/etc/oxidized/git-repos/default.git", @running=false, @last=#<OpenStruct start=2016-07-14 09:20:18 UTC, end=2016-07-14 09:20:24 UTC, status=:success, time=5.879200668>>, job=#<Oxidized::Job:0x00000003764f00 dead>, commitref="84d829beaf8edbd81888c2f076160f9279ac3b42", event=:post_store>
=> #<Oxidized::HookManager::HookContext node=#<Oxidized::Node:0x0000000392a0b0 @name="10.100.3.254", @ip="10.100.3.254", @group="net-cisco-backup", @input=[Oxidized::SSH], @output=Oxidized::Git, @model=#<IOS:0x000000036e6560 @node=#<Oxidized::Node:0x0000000392a0b0 ...>, @input=nil>, @auth={:username=>"bauxia", :password=>"Sd2D@D83LceP@33mu-2"}, @prompt=/^([\w.@()-]+[#>]\s?)$/, @vars=nil, @stats=#<Oxidized::Node::Stats:0x000000036e4f30 @stats={:success=>[{:start=>2016-07-14 09:20:18 UTC, :end=>2016-07-14 09:20:24 UTC, :time=>5.879200668}]}>, @retry=0, @repo="/etc/oxidized/git-repos/default.git", @running=false, @last=#<OpenStruct start=2016-07-14 09:20:18 UTC, end=2016-07-14 09:20:24 UTC, status=:success, time=5.879200668>>, job=#<Oxidized::Job:0x00000003764f00 dead>, commitref="84d829beaf8edbd81888c2f076160f9279ac3b42", event=:post_store>
[2] pry(#<GithubRepo>)> p ctx.node
#<Oxidized::Node:0x0000000392a0b0 @name="10.100.3.254", @ip="10.100.3.254", @group="net-cisco-backup", @input=[Oxidized::SSH], @output=Oxidized::Git, @model=#<IOS:0x000000036e6560 @node=#<Oxidized::Node:0x0000000392a0b0 ...>, @input=nil>, @auth={:username=>"bauxia", :password=>"Sd2D@D83LceP@33mu-2"}, @prompt=/^([\w.@()-]+[#>]\s?)$/, @vars=nil, @stats=#<Oxidized::Node::Stats:0x000000036e4f30 @stats={:success=>[{:start=>2016-07-14 09:20:18 UTC, :end=>2016-07-14 09:20:24 UTC, :time=>5.879200668}]}>, @retry=0, @repo="/etc/oxidized/git-repos/default.git", @running=false, @last=#<OpenStruct start=2016-07-14 09:20:18 UTC, end=2016-07-14 09:20:24 UTC, status=:success, time=5.879200668>>
=> #<Oxidized::Node:0x0000000392a0b0
 @auth={:username=>"bauxia", :password=>"Sd2D@D83LceP@33mu-2"},
 @group="net-cisco-backup",
 @input=[Oxidized::SSH],
 @ip="10.100.3.254",
 @last=#<OpenStruct start=2016-07-14 09:20:18 UTC, end=2016-07-14 09:20:24 UTC, status=:success, time=5.879200668>,
 @model=#<IOS:0x000000036e6560 @input=nil, @node=#<Oxidized::Node:0x0000000392a0b0 ...>>,
 @name="10.100.3.254",
 @output=Oxidized::Git,
 @prompt=/^([\w.@()-]+[#>]\s?)$/,
 @repo="/etc/oxidized/git-repos/default.git",
 @retry=0,
 @running=false,
 @stats=
  #<Oxidized::Node::Stats:0x000000036e4f30
   @stats={:success=>[{:start=>2016-07-14 09:20:18 UTC, :end=>2016-07-14 09:20:24 UTC, :time=>5.879200668}]}>,
 @vars=nil>
[3] pry(#<GithubRepo>)> p ctx.node.repo
"/etc/oxidized/git-repos/default.git"
=> "/etc/oxidized/git-repos/default.git"
[3] pry(#<GithubRepo>)> p ctx.node.repo
"/etc/oxidized/git-repos/default.git"
=> "/etc/oxidized/git-repos/default.git"
[4] pry(#<GithubRepo>)> p Oxidized.config.output.git.repo
"/etc/oxidized/git-repos/default.git"
=> "/etc/oxidized/git-repos/default.git"
[5] pry(#<GithubRepo>)> p Oxidized.config.output.git.repo.is_a?(String)
true
=> true
[6] pry(#<GithubRepo>)> p Oxidized.config.output.git.single_repo
#<Asetus::ConfigStruct:0x00000004c380f8 @key_to_s=nil, @cfg={}>
=> #<Asetus::ConfigStruct:0x00000004c380f8 @cfg={}, @key_to_s=nil>
[7] pry(#<GithubRepo>)>

Log

E, [2016-07-14T11:24:46.967258 #10032] ERROR -- : Hook push_to_remote_repo (#<GithubRepo:0x00000002052b78>) failed (#<Rugged::OSError: Failed to resolve path '/etc/oxidized/git-repos/default.git': No such file or directory>) for event :post_store
furhouse commented 8 years ago

After the change on line 177 from String to ::String, it seems that the hook is partially working. The testsrx config is pushed to the correct group repo (ens), but a device from another group (testprocurve, group lab) is also pushed to the ens repo.

testsrx:

root@debian-jessie:~/.config/oxidized# oxidized -d
I, [2016-07-14T09:38:59.116885 #3181]  INFO -- : Oxidized starting, running as pid 3181
D, [2016-07-14T09:38:59.230002 #3181] DEBUG -- : Hook "push_to_remote" registered GithubRepo for event :node_success
resolving DNS for testsrx.location.example.local...
resolving DNS for testubnt.location.example.local...
resolving DNS for testprocurve.location.example.local...
I, [2016-07-14T09:38:59.299717 #3181]  INFO -- : Loaded 3 nodes
D, [2016-07-14T09:38:59.629185 #3181] DEBUG -- : Jobs 0, Want: 1
Puma 2.16.0 starting...
* Min threads: 0, max threads: 16
* Environment: development
* Listening on tcp://0.0.0.0:80
D, [2016-07-14T09:39:00.861887 #3181] DEBUG -- : SSH: show configuration | display omit @ testsrx.location.example.local
D, [2016-07-14T09:39:02.379870 #3181] DEBUG -- : SSH: show version @ testsrx.location.example.local
D, [2016-07-14T09:39:03.321400 #3181] DEBUG -- : SSH: show chassis hardware @ testsrx.location.example.local

From: /var/lib/gems/2.1.0/gems/oxidized-0.14.3/lib/oxidized/hook/githubrepo.rb @ line 8 GithubRepo#run_hook:

     7: def run_hook(ctx)
 =>  8:   binding.pry
     9:   repo = Rugged::Repository.new(ctx.node.repo)
    10:   log "Pushing local repository(#{repo.path})..."
    11:   remote = repo.remotes['origin'] || repo.remotes.create('origin', remote_repo(ctx.node))
    12:   log "to remote: #{remote.url}"
    13:
    14:   fetch_and_merge_remote(repo)
    15:
    16:   remote.push([repo.head.name], credentials: credentials)
    17: end

[1] pry(#<GithubRepo>)> p ctx
#<Oxidized::HookManager::HookContext node=#<Oxidized::Node:0x00000001d724a8 @name="testsrx.location.example.local", @ip="10.53.14.230", @group="ens", @input=[Oxidized::SSH], @output=Oxidized::Git, @model=#<JunOS:0x00000001c4b390 @node=#<O
xidized::Node:0x00000001d724a8 ...>, @input=nil, @model="srx240h2">, @auth={:username=>"oxidized", :password=>"TxYSfgt5HgM6cH66R0s00vYY"}, @prompt=/^([\w.@-]+[#>]\s?)$/, @vars={:comware_cmdline=>nil}, @stats=#<Oxidized::Node::Stats:0x00000
001c4a7d8 @stats={:success=>[{:start=>2016-07-14 09:38:59 UTC, :end=>2016-07-14 09:39:03 UTC, :time=>4.30419841}]}>, @retry=0, @repo="/root/.config/oxidized/git-repos/default.git", @running=false, @last=#<OpenStruct start=2016-07-14 09:38:
59 UTC, end=2016-07-14 09:39:03 UTC, status=:success, time=4.30419841>>, job=#<Oxidized::Job:0x0000000297d388 dead>, event=:node_success>
=> #<Oxidized::HookManager::HookContext node=#<Oxidized::Node:0x00000001d724a8 @name="testsrx.location.example.local", @ip="10.53.14.230", @group="ens", @input=[Oxidized::SSH], @output=Oxidized::Git, @model=#<JunOS:0x00000001c4b390 @node=
#<Oxidized::Node:0x00000001d724a8 ...>, @input=nil, @model="srx240h2">, @auth={:username=>"oxidized", :password=>"TxYSfgt5HgM6cH66R0s00vYY"}, @prompt=/^([\w.@-]+[#>]\s?)$/, @vars={:comware_cmdline=>nil}, @stats=#<Oxidized::Node::Stats:0x00
000001c4a7d8 @stats={:success=>[{:start=>2016-07-14 09:38:59 UTC, :end=>2016-07-14 09:39:03 UTC, :time=>4.30419841}]}>, @retry=0, @repo="/root/.config/oxidized/git-repos/default.git", @running=false, @last=#<OpenStruct start=2016-07-14 09:
38:59 UTC, end=2016-07-14 09:39:03 UTC, status=:success, time=4.30419841>>, job=#<Oxidized::Job:0x0000000297d388 dead>, event=:node_success>
[2] pry(#<GithubRepo>)> p ctx.node
#<Oxidized::Node:0x00000001d724a8 @name="testsrx.location.example.local", @ip="10.53.14.230", @group="ens", @input=[Oxidized::SSH], @output=Oxidized::Git, @model=#<JunOS:0x00000001c4b390 @node=#<Oxidized::Node:0x00000001d724a8 ...>, @inpu
t=nil, @model="srx240h2">, @auth={:username=>"oxidized", :password=>"TxYSfgt5HgM6cH66R0s00vYY"}, @prompt=/^([\w.@-]+[#>]\s?)$/, @vars={:comware_cmdline=>nil}, @stats=#<Oxidized::Node::Stats:0x00000001c4a7d8 @stats={:success=>[{:start=>2016
-07-14 09:38:59 UTC, :end=>2016-07-14 09:39:03 UTC, :time=>4.30419841}]}>, @retry=0, @repo="/root/.config/oxidized/git-repos/default.git", @running=false, @last=#<OpenStruct start=2016-07-14 09:38:59 UTC, end=2016-07-14 09:39:03 UTC, statu
s=:success, time=4.30419841>>
=> #<Oxidized::Node:0x00000001d724a8
 @auth={:username=>"oxidized", :password=>"TxYSfgt5HgM6cH66R0s00vYY"},
 @group="ens",
 @input=[Oxidized::SSH],
 @ip="10.53.14.230",
 @last=#<OpenStruct start=2016-07-14 09:38:59 UTC, end=2016-07-14 09:39:03 UTC, status=:success, time=4.30419841>,
 @model=#<JunOS:0x00000001c4b390 @input=nil, @model="srx240h2", @node=#<Oxidized::Node:0x00000001d724a8 ...>>,
 @name="testsrx.location.example.local",
 @output=Oxidized::Git,
 @prompt=/^([\w.@-]+[#>]\s?)$/,
 @repo="/root/.config/oxidized/git-repos/default.git",
 @retry=0,
 @running=false,
 @stats=#<Oxidized::Node::Stats:0x00000001c4a7d8 @stats={:success=>[{:start=>2016-07-14 09:38:59 UTC, :end=>2016-07-14 09:39:03 UTC, :time=>4.30419841}]}>,
 @vars={:comware_cmdline=>nil}>
[3] pry(#<GithubRepo>)> p ctx.node.repo
"/root/.config/oxidized/git-repos/default.git"
=> "/root/.config/oxidized/git-repos/default.git"
[4] pry(#<GithubRepo>)> p Oxidized.config.output.git.repo
"/root/.config/oxidized/git-repos/default.git"
=> "/root/.config/oxidized/git-repos/default.git"
[5] pry(#<GithubRepo>)> p Oxidized.config.output.git.repo.is_a?(String)
true
=> true
[6] pry(#<GithubRepo>)> p Oxidized.config.output.git.single_repo
#<Asetus::ConfigStruct:0x0000000286cfc0 @key_to_s=nil, @cfg={}>
=> #<Asetus::ConfigStruct:0x0000000286cfc0 @cfg={}, @key_to_s=nil>
I, [2016-07-14T09:47:12.009634 #3181]  INFO -- : GithubRepo: Pushing local repository(/root/.config/oxidized/git-repos/default.git/)...
I, [2016-07-14T09:47:12.009885 #3181]  INFO -- : GithubRepo: to remote: git@git.example.com:oxidized/ens.git
D, [2016-07-14T09:47:12.010036 #3181] DEBUG -- : GithubRepo: Using ssh auth with key
D, [2016-07-14T09:47:12.803818 #3181] DEBUG -- : GithubRepo: {:total_objects=>0, :indexed_objects=>0, :received_objects=>0, :local_objects=>0, :total_deltas=>0, :indexed_deltas=>0, :received_bytes=>0}
D, [2016-07-14T09:47:12.804189 #3181] DEBUG -- : GithubRepo: nothing recieved after fetch

testprocurve:

D, [2016-07-14T09:47:21.896351 #3181] DEBUG -- : SSH: screen-length disable @ testprocurve.location.example.local
D, [2016-07-14T09:47:22.304295 #3181] DEBUG -- : SSH: undo terminal monitor @ testprocurve.location.example.local
D, [2016-07-14T09:47:22.710109 #3181] DEBUG -- : SSH: display version @ testprocurve.location.example.local
D, [2016-07-14T09:47:23.320822 #3181] DEBUG -- : SSH: display device @ testprocurve.location.example.local
D, [2016-07-14T09:47:23.728499 #3181] DEBUG -- : SSH: display current-configuration @ testprocurve.location.example.local
D, [2016-07-14T09:47:25.162520 #3181] DEBUG -- : SSH: quit @ testprocurve.location.example.local

From: /var/lib/gems/2.1.0/gems/oxidized-0.14.3/lib/oxidized/hook/githubrepo.rb @ line 8 GithubRepo#run_hook:

     7: def run_hook(ctx)
 =>  8:   binding.pry
     9:   repo = Rugged::Repository.new(ctx.node.repo)
    10:   log "Pushing local repository(#{repo.path})..."
    11:   remote = repo.remotes['origin'] || repo.remotes.create('origin', remote_repo(ctx.node))
    12:   log "to remote: #{remote.url}"
    13:
    14:   fetch_and_merge_remote(repo)
    15:
    16:   remote.push([repo.head.name], credentials: credentials)
    17: end

[1] pry(#<GithubRepo>)> p ctx
#<Oxidized::HookManager::HookContext node=#<Oxidized::Node:0x000000018bb1f8 @name="testprocurve.location.example.local", @ip="10.53.14.245", @group="lab", @input=[Oxidized::SSH], @output=Oxidized::Git, @model=#<Comware:0x000000016c6500 @n
ode=#<Oxidized::Node:0x000000018bb1f8 ...>, @input=nil>, @auth={:username=>"oxidized", :password=>"TxYSfgt5HgM6cH66R0s00vYY"}, @prompt=/^\0*(<[\w.-]+>)$/, @vars={:comware_cmdline=>"512900"}, @stats=#<Oxidized::Node::Stats:0x000000016c5bc8
@stats={:success=>[{:start=>2016-07-14 09:47:20 UTC, :end=>2016-07-14 09:47:25 UTC, :time=>4.717780876}]}>, @retry=0, @repo="/root/.config/oxidized/git-repos/default.git", @running=false, @last=#<OpenStruct start=2016-07-14 09:47:20 UTC, e
nd=2016-07-14 09:47:25 UTC, status=:success, time=4.717780876>>, job=#<Oxidized::Job:0x00000002c01078 dead>, event=:node_success>
=> #<Oxidized::HookManager::HookContext node=#<Oxidized::Node:0x000000018bb1f8 @name="testprocurve.location.example.local", @ip="10.53.14.245", @group="lab", @input=[Oxidized::SSH], @output=Oxidized::Git, @model=#<Comware:0x000000016c6500
 @node=#<Oxidized::Node:0x000000018bb1f8 ...>, @input=nil>, @auth={:username=>"oxidized", :password=>"TxYSfgt5HgM6cH66R0s00vYY"}, @prompt=/^\0*(<[\w.-]+>)$/, @vars={:comware_cmdline=>"512900"}, @stats=#<Oxidized::Node::Stats:0x000000016c5b
c8 @stats={:success=>[{:start=>2016-07-14 09:47:20 UTC, :end=>2016-07-14 09:47:25 UTC, :time=>4.717780876}]}>, @retry=0, @repo="/root/.config/oxidized/git-repos/default.git", @running=false, @last=#<OpenStruct start=2016-07-14 09:47:20 UTC
, end=2016-07-14 09:47:25 UTC, status=:success, time=4.717780876>>, job=#<Oxidized::Job:0x00000002c01078 dead>, event=:node_success>
[2] pry(#<GithubRepo>)> p ctx.node
#<Oxidized::Node:0x000000018bb1f8 @name="testprocurve.location.example.local", @ip="10.53.14.245", @group="lab", @input=[Oxidized::SSH], @output=Oxidized::Git, @model=#<Comware:0x000000016c6500 @node=#<Oxidized::Node:0x000000018bb1f8 ...>
, @input=nil>, @auth={:username=>"oxidized", :password=>"TxYSfgt5HgM6cH66R0s00vYY"}, @prompt=/^\0*(<[\w.-]+>)$/, @vars={:comware_cmdline=>"512900"}, @stats=#<Oxidized::Node::Stats:0x000000016c5bc8 @stats={:success=>[{:start=>2016-07-14 09:
47:20 UTC, :end=>2016-07-14 09:47:25 UTC, :time=>4.717780876}]}>, @retry=0, @repo="/root/.config/oxidized/git-repos/default.git", @running=false, @last=#<OpenStruct start=2016-07-14 09:47:20 UTC, end=2016-07-14 09:47:25 UTC, status=:succes
s, time=4.717780876>>
=> #<Oxidized::Node:0x000000018bb1f8
 @auth={:username=>"oxidized", :password=>"TxYSfgt5HgM6cH66R0s00vYY"},
 @group="lab",
 @input=[Oxidized::SSH],
 @ip="10.53.14.245",
 @last=#<OpenStruct start=2016-07-14 09:47:20 UTC, end=2016-07-14 09:47:25 UTC, status=:success, time=4.717780876>,
 @model=#<Comware:0x000000016c6500 @input=nil, @node=#<Oxidized::Node:0x000000018bb1f8 ...>>,
 @name="testprocurve.location.example.local",
 @output=Oxidized::Git,
 @prompt=/^\0*(<[\w.-]+>)$/,
 @repo="/root/.config/oxidized/git-repos/default.git",
 @retry=0,
 @running=false,
 @stats=#<Oxidized::Node::Stats:0x000000016c5bc8 @stats={:success=>[{:start=>2016-07-14 09:47:20 UTC, :end=>2016-07-14 09:47:25 UTC, :time=>4.717780876}]}>,
 @vars={:comware_cmdline=>"512900"}>
[3] pry(#<GithubRepo>)> p ctx.node.repo
"/root/.config/oxidized/git-repos/default.git"
=> "/root/.config/oxidized/git-repos/default.git"
[4] pry(#<GithubRepo>)> p Oxidized.config.output.git.repo
"/root/.config/oxidized/git-repos/default.git"
=> "/root/.config/oxidized/git-repos/default.git"
[5] pry(#<GithubRepo>)> p Oxidized.config.output.git.repo.is_a?(String)
true
=> true
[7] pry(#<GithubRepo>)> p Oxidized.config.output.git.single_repo
#<Asetus::ConfigStruct:0x0000000286cfc0 @key_to_s=nil, @cfg={}>
=> #<Asetus::ConfigStruct:0x0000000286cfc0 @cfg={}, @key_to_s=nil>

I, [2016-07-14T09:49:00.073149 #3181]  INFO -- : GithubRepo: Pushing local repository(/root/.config/oxidized/git-repos/default.git/)...
I, [2016-07-14T09:49:00.073255 #3181]  INFO -- : GithubRepo: to remote: git@git.example.com:oxidized/ens.git
D, [2016-07-14T09:49:00.730730 #3181] DEBUG -- : GithubRepo: {:total_objects=>0, :indexed_objects=>0, :received_objects=>0, :local_objects=>0, :total_deltas=>0, :indexed_deltas=>0, :received_bytes=>0}
D, [2016-07-14T09:49:00.731283 #3181] DEBUG -- : GithubRepo: nothing recieved after fetch
totosh commented 8 years ago

remote_repo.is_a?(::String) match the condition so remote_repo will be Oxidized.config.output.git.repo. That we don't want.

remote_repo.is_a?(String) was the good syntax.


From: /usr/local/rvm/gems/ruby-2.1.8/gems/oxidized-0.15.0/lib/oxidized/node.rb @ line 180 Oxidized::Node#resolve_repo:

    175:     def resolve_repo
    176:       remote_repo = Oxidized.config.output.git.repo
    177:       if Oxidized.config.output.git.single_repo? || @group.nil? || remote_repo.is_a?(String)
    178:         remote_repo
    179:       else
 => 180:        binding.pry
    181:         remote_repo[@group]
    182:       end
    183:     end

[1] pry(#<Oxidized::Node>)> Oxidized.config.output.git.repo
=> "/etc/oxidized/git-repos/default.git"
[2] pry(#<Oxidized::Node>)> remote_repo
=> "/etc/oxidized/git-repos/default.git"
[3] pry(#<Oxidized::Node>)> @group
=> "net-cisco-backup"
[4] pry(#<Oxidized::Node>)> remote_repo[@group]
=> nil
[5] pry(#<Oxidized::Node>)>
ytti commented 8 years ago

@totosh do you still have group name in the output: git: repo? It looks like it's just string now.

Try:

  1. ::String
  2. have group name in the output: git: repo: xxx:
totosh commented 8 years ago

I have.

  default: git
  git:
    user:  #####
    email: #####
    repo: "/etc/oxidized/git-repos/default.git"

With::String, the condition will be true if i keep repo in the output: git. If i remove it, the condition will be false.

Output with ::String and no repo in the output: git

From: /usr/local/rvm/gems/ruby-2.1.8/gems/oxidized-0.15.0/lib/oxidized/node.rb @ line 180 Oxidized::Node#resolve_repo:

    175:     def resolve_repo
    176:       remote_repo = Oxidized.config.output.git.repo
    177:       if Oxidized.config.output.git.single_repo? || @group.nil? || remote_repo.is_a?(::String)
    178:         remote_repo
    179:       else
 => 180:        binding.pry
    181:         remote_repo[@group]
    182:       end
    183:     end

[1] pry(#<Oxidized::Node>)> @group
=> "net-cisco-backup"
[2] pry(#<Oxidized::Node>)> remote_repo[@group]
=> #<Asetus::ConfigStruct:0x000000039d68b0 @cfg={}, @key_to_s=nil>
[3] pry(#<Oxidized::Node>)> remote_repo
=> #<Asetus::ConfigStruct:0x00000003d943a8
 @cfg={"net-cisco-backup"=>#<Asetus::ConfigStruct:0x000000039d68b0 @cfg={}, @key_to_s=nil>},
 @key_to_s=nil>

I think, it's better to keep a default repo.

ytti commented 8 years ago

::String is definitely correct. If it is ::String you should use just the plain string argument.

If it's not ::String, then w should ask for the hash key, which is group, which should then return ::String.

What I don't understand in the output, why is remote_repo in your instance claiming that there is 'net-cisco-backup' configured, but does not actually have value?

Both of these should work

output:
  git:
    repo: string

and

output:
  git:
   repo:
      group: string

the first should choose the first branch the second should choose the else branch.

It's not now at all obvious to me what was your config whtn remote_repo had that output. It looks like your config and your output is from different runtimes.

totosh commented 8 years ago

With the second syntax group: string I get a crash and local repo don't work.

  git:
    repo:
      group: default

Crash log

--------------------------------------------------
2016-07-14 11:50:56 UTC
wrong argument type Asetus::ConfigStruct (expected String) [TypeError]
--------------------------------------------------
/usr/local/rvm/gems/ruby-2.1.8/gems/oxidized-0.15.0/lib/oxidized/output/git.rb:176:in `new'
/usr/local/rvm/gems/ruby-2.1.8/gems/oxidized-0.15.0/lib/oxidized/output/git.rb:176:in `update'
/usr/local/rvm/gems/ruby-2.1.8/gems/oxidized-0.15.0/lib/oxidized/output/git.rb:57:in `store'
/usr/local/rvm/gems/ruby-2.1.8/gems/oxidized-0.15.0/lib/oxidized/worker.rb:47:in `process'
/usr/local/rvm/gems/ruby-2.1.8/gems/oxidized-0.15.0/lib/oxidized/worker.rb:14:in `block in work'
/usr/local/rvm/gems/ruby-2.1.8/gems/oxidized-0.15.0/lib/oxidized/worker.rb:14:in `each'
/usr/local/rvm/gems/ruby-2.1.8/gems/oxidized-0.15.0/lib/oxidized/worker.rb:14:in `work'
/usr/local/rvm/gems/ruby-2.1.8/gems/oxidized-0.15.0/lib/oxidized/core.rb:36:in `run'
/usr/local/rvm/gems/ruby-2.1.8/gems/oxidized-0.15.0/lib/oxidized/core.rb:28:in `initialize'
/usr/local/rvm/gems/ruby-2.1.8/gems/oxidized-0.15.0/lib/oxidized/core.rb:4:in `new'
/usr/local/rvm/gems/ruby-2.1.8/gems/oxidized-0.15.0/lib/oxidized/core.rb:4:in `new'
/usr/local/rvm/gems/ruby-2.1.8/gems/oxidized-0.15.0/lib/oxidized/cli.rb:12:in `run'
/usr/local/rvm/gems/ruby-2.1.8/gems/oxidized-0.15.0/bin/oxidized:9:in `<top (required)>'
/usr/local/rvm/gems/ruby-2.1.8/bin/oxidized:23:in `load'
/usr/local/rvm/gems/ruby-2.1.8/bin/oxidized:23:in `<main>'
/usr/local/rvm/gems/ruby-2.1.8/bin/ruby_executable_hooks:15:in `eval'
/usr/local/rvm/gems/ruby-2.1.8/bin/ruby_executable_hooks:15:in `<main>'
--------------------------------------------------
ytti commented 8 years ago

Did you use 'group' verbatim, or the actual group name? I mean the actual group name.

Both of them should work. If it's string, then it should use just same repo for all, if ti's not string, then the hash key should be the group name.

If you prefer to share like 'tmate' session or what ever, to recreate this in your network where I can take a look at it, that would be fine.

totosh commented 8 years ago

I was mistaken :) I wrote 'group' verbatim. If I use this, it work well : 

output:
  git:
    repo:
      net-cisco-backup: "/etc/oxidized/git-repos/net-cisco-backup.git"

Therefore, group git must be specify in the output section whether we want to use remote git by group even if by default local git is automatically created by group whitout specify it in the output section .

Valid config for remote git :

output:
  default: git
  git:
    user: #####
    email: #####
    repo: 
      groupname1: "/etc/oxidized/git-repos/groupname1.git"
      groupname2: "/etc/oxidized/git-repos/groupname2.git"
      groupname3: "/etc/oxidized/git-repos/groupname3.git"
source:
    map:
      name: address
      model: model
      group: groupname
hooks:
  push_to_remote_repo:
    type: githubrepo
    remote_repo:
      groupname1: "git@gitlab.domain.lan:10022/oxidized/groupname1.git"
      groupname2: "git@gitlab.domain.lan:10022/oxidized/groupname2.git"
      groupname3: "git@gitlab.domain.lan:10022/oxidized/groupname3.git"
    events: [post_store]
    publickey: /home/oxidized/.ssh/id_rsa.pub
    privatekey: /home/oxidized/.ssh/id_rsa

Thank you !

ytti commented 8 years ago

Both should work. With ::String change it should have worked without the hash. Can you test this? Because I really think the hook is wrong comparing to String (Which is actually Oxidized::String).

furhouse commented 8 years ago

I was testing with the same notation for the output/hooks, but the first commit seems to fail because the repository hasn't been created yet. The second device in the same group works fine. After the first interval, all tasks seem complete.

This is with ::String by the way.

root@debian-jessie:~/.config/oxidized# rm -rf git-repos/*
root@debian-jessie:~/.config/oxidized# oxidized -d                                                                                                                                                                                             
I, [2016-07-14T13:31:11.223801 #28714]  INFO -- : Oxidized starting, running as pid 28714
D, [2016-07-14T13:31:11.236331 #28714] DEBUG -- : Hook "push_to_remote" registered GithubRepo for event :node_success
I, [2016-07-14T13:31:11.237595 #28714]  INFO -- : lib/oxidized/nodes.rb: Loading nodes
D, [2016-07-14T13:31:11.238174 #28714] DEBUG -- : resolving DNS for testsrx.location.example.local...
D, [2016-07-14T13:31:11.289699 #28714] DEBUG -- : lib/oxidized/node.rb: Loading model "junos"
D, [2016-07-14T13:31:11.292464 #28714] DEBUG -- : lib/oxidized/model/model.rb Added all to the commands list
D, [2016-07-14T13:31:11.292924 #28714] DEBUG -- : lib/oxidized/model/model.rb Added secret to the commands list
D, [2016-07-14T13:31:11.293300 #28714] DEBUG -- : lib/oxidized/model/model.rb Added show configuration | display omit to the commands list
D, [2016-07-14T13:31:11.293698 #28714] DEBUG -- : lib/oxidized/model/model.rb Added show version to the commands list
D, [2016-07-14T13:31:11.293961 #28714] DEBUG -- : lib/oxidized/model/model.rb Added show chassis hardware to the commands list
D, [2016-07-14T13:31:11.294833 #28714] DEBUG -- : resolving DNS for testubnt.location.example.local...
D, [2016-07-14T13:31:11.295418 #28714] DEBUG -- : lib/oxidized/node.rb: Loading model "edgeos"
D, [2016-07-14T13:31:11.297729 #28714] DEBUG -- : lib/oxidized/model/model.rb Added all to the commands list
D, [2016-07-14T13:31:11.298088 #28714] DEBUG -- : lib/oxidized/model/model.rb Added secret to the commands list
D, [2016-07-14T13:31:11.298436 #28714] DEBUG -- : lib/oxidized/model/model.rb Added show configuration | no-more to the commands list
D, [2016-07-14T13:31:11.299348 #28714] DEBUG -- : resolving DNS for testprocurve.location.example.local...
D, [2016-07-14T13:31:11.300027 #28714] DEBUG -- : lib/oxidized/node.rb: Loading model "comware"
D, [2016-07-14T13:31:11.302504 #28714] DEBUG -- : lib/oxidized/model/model.rb Added all to the commands list
D, [2016-07-14T13:31:11.302894 #28714] DEBUG -- : lib/oxidized/model/model.rb Added display version to the commands list
D, [2016-07-14T13:31:11.303253 #28714] DEBUG -- : lib/oxidized/model/model.rb Added display device to the commands list
D, [2016-07-14T13:31:11.303594 #28714] DEBUG -- : lib/oxidized/model/model.rb Added display current-configuration to the commands list
I, [2016-07-14T13:31:11.304441 #28714]  INFO -- : lib/oxidized/nodes.rb: Loaded 3 nodes
D, [2016-07-14T13:31:11.614574 #28714] DEBUG -- : lib/oxidized/core.rb: Starting the worker...
D, [2016-07-14T13:31:11.615039 #28714] DEBUG -- : lib/oxidized/worker.rb: Jobs 0, Want: 1
D, [2016-07-14T13:31:11.615700 #28714] DEBUG -- : lib/oxidized/worker.rb: Added testsrx.location.example.local to the job queue
D, [2016-07-14T13:31:11.616142 #28714] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2016-07-14T13:31:11.616596 #28714] DEBUG -- : lib/oxidized/job.rb: Starting fetching process for testsrx.location.example.local at 2016-07-14 13:31:11 UTC
D, [2016-07-14T13:31:11.617252 #28714] DEBUG -- : lib/oxidized/input/ssh.rb: Connecting to testsrx.location.example.local
Puma 2.16.0 starting...
* Min threads: 0, max threads: 16
* Environment: development
* Listening on tcp://0.0.0.0:80
D, [2016-07-14T13:31:12.617646 #28714] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2016-07-14T13:31:12.837381 #28714] DEBUG -- : lib/oxidized/input/cli.rb: Running post_login commands at testsrx.location.example.local
D, [2016-07-14T13:31:12.837954 #28714] DEBUG -- : lib/oxidized/model/model.rb Collecting commands' outputs
D, [2016-07-14T13:31:12.838596 #28714] DEBUG -- : lib/oxidized/model/model.rb Executing show configuration | display omit
D, [2016-07-14T13:31:12.838991 #28714] DEBUG -- : lib/oxidized/input/ssh.rb show configuration | display omit @ testsrx.location.example.local with expect: /^([\w.@-]+[#>]\s?)$/
D, [2016-07-14T13:31:13.619187 #28714] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2016-07-14T13:31:14.619538 #28714] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2016-07-14T13:31:14.653349 #28714] DEBUG -- : lib/oxidized/model/model.rb Executing show version
D, [2016-07-14T13:31:14.653806 #28714] DEBUG -- : lib/oxidized/input/ssh.rb show version @ testsrx.location.example.local with expect: /^([\w.@-]+[#>]\s?)$/
D, [2016-07-14T13:31:15.606110 #28714] DEBUG -- : lib/oxidized/model/model.rb Executing show chassis hardware
D, [2016-07-14T13:31:15.606539 #28714] DEBUG -- : lib/oxidized/input/ssh.rb show chassis hardware @ testsrx.location.example.local with expect: /^([\w.@-]+[#>]\s?)$/
D, [2016-07-14T13:31:15.620073 #28714] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2016-07-14T13:31:16.526052 #28714] DEBUG -- : lib/oxidized/input/cli.rb Running pre_logout commands at testsrx.location.example.local
D, [2016-07-14T13:31:16.526963 #28714] DEBUG -- : lib/oxidized/node.rb: Oxidized::SSH ran for testsrx.location.example.local successfully
D, [2016-07-14T13:31:16.527050 #28714] DEBUG -- : lib/oxidized/job.rb: Config fetched for testsrx.location.example.local at 2016-07-14 13:31:16 UTC
E, [2016-07-14T13:31:16.621280 #28714] ERROR -- : Hook push_to_remote (#<GithubRepo:0x000000020ea0e0>) failed (#<Rugged::OSError: Failed to resolve path '/root/.config/oxidized/git-repos/ens.git': No such file or directory>) for event :nod
e_success
I, [2016-07-14T13:31:16.626135 #28714]  INFO -- : Configuration updated for ens/testsrx.location.example.local
D, [2016-07-14T13:31:16.626671 #28714] DEBUG -- : lib/oxidized/worker.rb: Jobs 0, Want: 1
D, [2016-07-14T13:31:16.627177 #28714] DEBUG -- : lib/oxidized/worker.rb: Added testubnt.location.example.local to the job queue
D, [2016-07-14T13:31:16.627434 #28714] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2016-07-14T13:31:16.627853 #28714] DEBUG -- : lib/oxidized/job.rb: Starting fetching process for testubnt.location.example.local at 2016-07-14 13:31:16 UTC
D, [2016-07-14T13:31:16.628224 #28714] DEBUG -- : lib/oxidized/input/ssh.rb: Connecting to testubnt.location.example.local
D, [2016-07-14T13:31:17.627966 #28714] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2016-07-14T13:31:17.750910 #28714] DEBUG -- : lib/oxidized/input/ssh.rb: expecting /\@.*?\:~\$\s/ at testubnt.location.example.local
D, [2016-07-14T13:31:18.628592 #28714] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2016-07-14T13:31:18.862524 #28714] DEBUG -- : lib/oxidized/input/cli.rb: Running post_login commands at testubnt.location.example.local
D, [2016-07-14T13:31:18.862980 #28714] DEBUG -- : lib/oxidized/model/model.rb Collecting commands' outputs
D, [2016-07-14T13:31:18.863378 #28714] DEBUG -- : lib/oxidized/model/model.rb Executing show configuration | no-more
D, [2016-07-14T13:31:18.863714 #28714] DEBUG -- : lib/oxidized/input/ssh.rb show configuration | no-more @ testubnt.location.example.local with expect: /\@.*?\:~\$\s/
D, [2016-07-14T13:31:18.864215 #28714] DEBUG -- : lib/oxidized/input/ssh.rb: expecting /\@.*?\:~\$\s/ at testubnt.location.example.local
D, [2016-07-14T13:31:19.629157 #28714] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2016-07-14T13:31:19.797268 #28714] DEBUG -- : lib/oxidized/input/cli.rb Running pre_logout commands at testubnt.location.example.local
D, [2016-07-14T13:31:19.797811 #28714] DEBUG -- : lib/oxidized/input/ssh.rb exit @ testubnt.location.example.local with expect: nil
D, [2016-07-14T13:31:19.823522 #28714] DEBUG -- : lib/oxidized/node.rb: Oxidized::SSH ran for testubnt.location.example.local successfully
D, [2016-07-14T13:31:19.824120 #28714] DEBUG -- : lib/oxidized/job.rb: Config fetched for testubnt.location.example.local at 2016-07-14 13:31:19 UTC
I, [2016-07-14T13:31:20.630908 #28714]  INFO -- : GithubRepo: Pushing local repository(/root/.config/oxidized/git-repos/ens.git/)...
I, [2016-07-14T13:31:20.631971 #28714]  INFO -- : GithubRepo: to remote: git@git.example.com:oxidized/ens.git
D, [2016-07-14T13:31:20.632083 #28714] DEBUG -- : GithubRepo: Using ssh auth with key
D, [2016-07-14T13:31:21.184498 #28714] DEBUG -- : GithubRepo: {:total_objects=>6, :indexed_objects=>6, :received_objects=>6, :local_objects=>0, :total_deltas=>1, :indexed_deltas=>1, :received_bytes=>2923}
D, [2016-07-14T13:31:21.185789 #28714] DEBUG -- : GithubRepo: merging fetched branch origin/master
D, [2016-07-14T13:31:22.236320 #28714] DEBUG -- : lib/oxidized/worker.rb: Jobs 0, Want: 1
D, [2016-07-14T13:31:22.236810 #28714] DEBUG -- : lib/oxidized/worker.rb: Added testprocurve.location.example.local to the job queue
D, [2016-07-14T13:31:22.237215 #28714] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2016-07-14T13:31:22.237616 #28714] DEBUG -- : lib/oxidized/job.rb: Starting fetching process for testprocurve.location.example.local at 2016-07-14 13:31:22 UTC
D, [2016-07-14T13:31:22.238110 #28714] DEBUG -- : lib/oxidized/input/ssh.rb: Connecting to testprocurve.location.example.local
D, [2016-07-14T13:31:22.632615 #28714] DEBUG -- : lib/oxidized/input/ssh.rb: expecting /^\0*(<[\w.-]+>)$/ at testprocurve.location.example.local
D, [2016-07-14T13:31:23.237746 #28714] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2016-07-14T13:31:23.643119 #28714] DEBUG -- : lib/oxidized/input/cli.rb: Running post_login commands at testprocurve.location.example.local
D, [2016-07-14T13:31:23.643872 #28714] DEBUG -- : lib/oxidized/input/cli.rb: Running post_login command: nil, block: #<Proc:0x00000003246a50@/var/lib/gems/2.1.0/gems/oxidized-0.15.0/lib/oxidized/model/comware.rb:31> at testprocurve.ensched
e.example.local
D, [2016-07-14T13:31:23.644477 #28714] DEBUG -- : lib/oxidized/input/cli.rb: Running post_login command: "screen-length disable", block: nil at testprocurve.location.example.local
D, [2016-07-14T13:31:23.644805 #28714] DEBUG -- : lib/oxidized/input/ssh.rb screen-length disable @ testprocurve.location.example.local with expect: /^\0*(<[\w.-]+>)$/
D, [2016-07-14T13:31:23.645194 #28714] DEBUG -- : lib/oxidized/input/ssh.rb: expecting /^\0*(<[\w.-]+>)$/ at testprocurve.location.example.local
D, [2016-07-14T13:31:24.051248 #28714] DEBUG -- : lib/oxidized/input/cli.rb: Running post_login command: "undo terminal monitor", block: nil at testprocurve.location.example.local
D, [2016-07-14T13:31:24.051646 #28714] DEBUG -- : lib/oxidized/input/ssh.rb undo terminal monitor @ testprocurve.location.example.local with expect: /^\0*(<[\w.-]+>)$/
D, [2016-07-14T13:31:24.052737 #28714] DEBUG -- : lib/oxidized/input/ssh.rb: expecting /^\0*(<[\w.-]+>)$/ at testprocurve.location.example.local
D, [2016-07-14T13:31:24.237972 #28714] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2016-07-14T13:31:24.457946 #28714] DEBUG -- : lib/oxidized/model/model.rb Collecting commands' outputs
D, [2016-07-14T13:31:24.458475 #28714] DEBUG -- : lib/oxidized/model/model.rb Executing display version
D, [2016-07-14T13:31:24.458986 #28714] DEBUG -- : lib/oxidized/input/ssh.rb display version @ testprocurve.location.example.local with expect: /^\0*(<[\w.-]+>)$/
D, [2016-07-14T13:31:24.459508 #28714] DEBUG -- : lib/oxidized/input/ssh.rb: expecting /^\0*(<[\w.-]+>)$/ at testprocurve.location.example.local
D, [2016-07-14T13:31:25.068168 #28714] DEBUG -- : lib/oxidized/model/model.rb Executing display device
D, [2016-07-14T13:31:25.068724 #28714] DEBUG -- : lib/oxidized/input/ssh.rb display device @ testprocurve.location.example.local with expect: /^\0*(<[\w.-]+>)$/
D, [2016-07-14T13:31:25.069274 #28714] DEBUG -- : lib/oxidized/input/ssh.rb: expecting /^\0*(<[\w.-]+>)$/ at testprocurve.location.example.local
D, [2016-07-14T13:31:25.238150 #28714] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2016-07-14T13:31:25.477185 #28714] DEBUG -- : lib/oxidized/model/model.rb Executing display current-configuration
D, [2016-07-14T13:31:25.477624 #28714] DEBUG -- : lib/oxidized/input/ssh.rb display current-configuration @ testprocurve.location.example.local with expect: /^\0*(<[\w.-]+>)$/
D, [2016-07-14T13:31:25.478221 #28714] DEBUG -- : lib/oxidized/input/ssh.rb: expecting /^\0*(<[\w.-]+>)$/ at testprocurve.location.example.local
D, [2016-07-14T13:31:26.238437 #28714] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2016-07-14T13:31:27.107942 #28714] DEBUG -- : lib/oxidized/input/cli.rb Running pre_logout commands at testprocurve.location.example.local
D, [2016-07-14T13:31:27.108798 #28714] DEBUG -- : lib/oxidized/input/ssh.rb quit @ testprocurve.location.example.local with expect: nil
D, [2016-07-14T13:31:27.140738 #28714] DEBUG -- : lib/oxidized/node.rb: Oxidized::SSH ran for testprocurve.location.example.local successfully
D, [2016-07-14T13:31:27.141211 #28714] DEBUG -- : lib/oxidized/job.rb: Config fetched for testprocurve.location.example.local at 2016-07-14 13:31:27 UTC
E, [2016-07-14T13:31:27.239141 #28714] ERROR -- : Hook push_to_remote (#<GithubRepo:0x000000020ea0e0>) failed (#<Rugged::OSError: Failed to resolve path '/root/.config/oxidized/git-repos/lab.git': No such file or directory>) for event :nod
e_success
I, [2016-07-14T13:31:27.242337 #28714]  INFO -- : Configuration updated for lab/testprocurve.location.example.local
D, [2016-07-14T13:31:27.243171 #28714] DEBUG -- : lib/oxidized/worker.rb: Jobs 0, Want: 1
output:
  default: git
  git:
    user: oxidized
    email: oxidized@example.com
    repo: "/root/.config/oxidized/git-repos/default.git"

to

output:
  default: git
  git:
    user: oxidized
    email: oxidized@example.com
    repo:
      ens: "/root/.config/oxidized/git-repos/ens.git"
      lab: "/root/.config/oxidized/git-repos/lab.git"

Changing that output hash seems to break the webinterface though, fetching the config of any node returns node not found and 

2016-07-14 13:41:27 - ArgumentError - wrong number of arguments (0 for 1+):
        /var/lib/gems/2.1.0/gems/oxidized-web-0.7.1/lib/oxidized/web/views/versions.haml:24:in `count'
        /var/lib/gems/2.1.0/gems/oxidized-web-0.7.1/lib/oxidized/web/views/versions.haml:24:in `block in singleton class'
        /var/lib/gems/2.1.0/gems/oxidized-web-0.7.1/lib/oxidized/web/views/versions.haml:-9:in `instance_eval'
        /var/lib/gems/2.1.0/gems/oxidized-web-0.7.1/lib/oxidized/web/views/versions.haml:-9:in `singleton class'
        /var/lib/gems/2.1.0/gems/oxidized-web-0.7.1/lib/oxidized/web/views/versions.haml:-11:in `__tilt_19397780'
        /var/lib/gems/2.1.0/gems/tilt-2.0.5/lib/tilt/template.rb:167:in `call'
        /var/lib/gems/2.1.0/gems/tilt-2.0.5/lib/tilt/template.rb:167:in `evaluate'
        /var/lib/gems/2.1.0/gems/tilt-2.0.5/lib/tilt/haml.rb:19:in `evaluate'
        /var/lib/gems/2.1.0/gems/tilt-2.0.5/lib/tilt/template.rb:102:in `render'
        /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:823:in `render'
        /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:677:in `haml'
        /var/lib/gems/2.1.0/gems/oxidized-web-0.7.1/lib/oxidized/web/webapp.rb:235:in `out'
        /var/lib/gems/2.1.0/gems/oxidized-web-0.7.1/lib/oxidized/web/webapp.rb:162:in `block in <class:WebApp>'
        /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1611:in `call'
        /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1611:in `block in compile!'
        /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in `[]'
        /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in `block (3 levels) in route!'
        /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:994:in `route_eval'
        /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in `block (2 levels) in route!'
        /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1015:in `block in process_route'
        /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1013:in `catch'
        /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1013:in `process_route'
        /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:973:in `block in route!'
        /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:972:in `each'
        /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:972:in `route!'
        /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1085:in `block in dispatch!'
        /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `block in invoke'
        /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `catch'
        /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `invoke'
        /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1082:in `dispatch!'
        /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:907:in `block in call!'
        /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `block in invoke'
        /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `catch'
        /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `invoke'
        /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:907:in `call!'
        /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:895:in `call'
        /var/lib/gems/2.1.0/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call'
        /var/lib/gems/2.1.0/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call'
        /var/lib/gems/2.1.0/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call'
        /var/lib/gems/2.1.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
        /var/lib/gems/2.1.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
        /var/lib/gems/2.1.0/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
        /var/lib/gems/2.1.0/gems/rack-1.6.4/lib/rack/nulllogger.rb:9:in `call'
        /var/lib/gems/2.1.0/gems/rack-1.6.4/lib/rack/head.rb:13:in `call'
        /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/show_exceptions.rb:25:in `call'
        /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:182:in `call'
        /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:2013:in `call'
        /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1487:in `block in call'
        /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1787:in `synchronize'
        /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1487:in `call'
        /var/lib/gems/2.1.0/gems/rack-1.6.4/lib/rack/urlmap.rb:66:in `block in call'
        /var/lib/gems/2.1.0/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `each'
        /var/lib/gems/2.1.0/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `call'
        /var/lib/gems/2.1.0/gems/rack-1.6.4/lib/rack/builder.rb:153:in `call'
        /var/lib/gems/2.1.0/gems/puma-2.16.0/lib/puma/server.rb:557:in `handle_request'
        /var/lib/gems/2.1.0/gems/puma-2.16.0/lib/puma/server.rb:404:in `process_client'
        /var/lib/gems/2.1.0/gems/puma-2.16.0/lib/puma/server.rb:270:in `block in run'
        /var/lib/gems/2.1.0/gems/puma-2.16.0/lib/puma/thread_pool.rb:106:in `call'
        /var/lib/gems/2.1.0/gems/puma-2.16.0/lib/puma/thread_pool.rb:106:in `block in spawn_thread'
danilopopeye commented 8 years ago

I think the problem its that the git.repo is using a String and the githubhook is using a Hash in the config exemples given. The git config really works with both options, but I suspect both have to be in the same format for the hook to work.

furhouse commented 8 years ago

But how would I provide the git.repo w/ groups as a string then?

output:
  default: git
  git:
    user: oxidized
    email: oxidized@example.com
    repo: "/root/.config/oxidized/git-repos/devices.git"

+

hooks:
  push_to_remote:
    type: githubrepo
    remote_repo: git@git.example.com:oxidized/devices.git
    events: [node_success]
    publickey: /root/.ssh/id_rsa.pub
    privatekey: /root/.ssh/id_rsa

Result: 

ERROR -- : Hook push_to_remote (#<GithubRepo:0x000000022660e0>) failed (#<Rugged::OSError: Failed to resolve path '/root/.config/oxidized/git-repos/devices.git': No such file or directory>) for event :node_success

Changing git.repo & push_to_remote.remote_repo to a hash:

output:
  default: git
  git:
    user: oxidized
    email: oxidized@example.com
    repo:
      ens: "/root/.config/oxidized/git-repos/ens.git"
      lab: "/root/.config/oxidized/git-repos/lab.git"

+

hooks:
  push_to_remote:
    type: githubrepo
    remote_repo:
      ens: git@git.example.com:oxidized/ens.git
      lab: git@git.example.com:oxidized/lab.git
    events: [node_success]
    publickey: /root/.ssh/id_rsa.pub
    privatekey: /root/.ssh/id_rsa

Result:

Now the webinterface won't work (GET @ /node/fetch/[GROUP]/[NODE] -> node not found, GET @ /node/version?node_full=[GROUP]/[NODE] -> wrong number of arguments (0 for 1+))

furhouse commented 8 years ago

@danilopopeye The issues with the web interface have been resolved after rebuilding the gem from the master branch after the merge.

danilopopeye commented 8 years ago

@furhouse glad to have helped :smile: :tada:

@totosh can you confirm it worked for you too, please?

danilopopeye commented 8 years ago

@totosh ping

totosh commented 8 years ago

Yes, problem solved ! :) Thx