ytti / oxidized

Oxidized is a network device configuration backup tool. It's a RANCID replacement!
Apache License 2.0
2.82k stars 931 forks source link

Hook push_to_remote (#<GithubRepo:0x00000002b9b958>) failed (#<TypeError: wrong argument type Fixnum (expected String)>) for event :post_store #1193

Closed vinesman closed 6 years ago

vinesman commented 6 years ago

Having an odd issue with the push_to_remote

Here are configs

---
username: username
password: password
model: cisco
interval: 3600
use_syslog: true
debug: true
threads: 30
timeout: 200
retries: 3
prompt: !ruby/regexp /([\w.@-]+[#>]\s?)$/
rest: 127.0.0.1:8888
next_adds_job: false
vars: {}
groups: {}
models: {}
pid: "/home/oxidized/.config/oxidized/pid"
input:
  default: ssh, telnet
  debug: true
  ssh:
    secure: false
output:
  default: git
  git:
    user: nms
    email: noreply1@domain.com
    repo: "/home/oxidized/.config/oxidixed/.git"
source:
  default: csv
  csv:
    file: "/home/oxidized/.config/oxidized/router.db"
    delimiter: !ruby/regexp /:/
    map:
      name: 0
      model: 1
      username: 2
      password: 3
    vars_map:
      enable: 4
      ssh_proxy: 5
    gpg: false
model_map:
  cisco: ios
  juniper: junos
  dell: powerconnect
hooks:
  push_to_remote:
    type: githubrepo
    events: [post_store]
    remote_repo: https://git01.windows.domain.com/Monitoring/networkconfigs.git
    username: nms
    password: 1122334455

Error

Feb 22 16:13:37 nms01 oxidized[49098]: Configuration updated for /Nstar-Switch-SP2
Feb 22 16:13:37 nms01 oxidized[49098]: GithubRepo: Pushing local repository(/home/oxidized/.config/oxidixed/)...
Feb 22 16:13:37 nms01 oxidized[49098]: GithubRepo: to remote: https://git01.windows.domain.com/Monitoring/networkconfigs.git
Feb 22 16:13:37 nms01 oxidized[49098]: GithubRepo: {:total_objects=>0, :indexed_objects=>0, :received_objects=>0, :local_objects=>0, :total_deltas=>0, :indexed_deltas=>0, :received_bytes=>0}
Feb 22 16:13:37 nms01 oxidized[49098]: GithubRepo: nothing recieved after fetch
Feb 22 16:13:37 nms01 oxidized[49098]: Hook push_to_remote (#<GithubRepo:0x00000002b9b958>) failed (#<TypeError: wrong argument type Fixnum (expected String)>) for event :post_store

Checks on Git

root@nms01:/home/oxidized/.config/oxidized# git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   config
        modified:   logs/192.168.99.1-ssh
        modified:   logs/192.168.99.10-telnet
        modified:   logs/192.168.99.122-telnet
        modified:   logs/192.168.99.16-ssh
        modified:   logs/192.168.99.197-ssh
        modified:   logs/192.168.99.28-ssh
        modified:   logs/192.168.99.4-telnet
        modified:   logs/192.168.99.5-telnet
        modified:   logs/192.168.99.69-ssh
        modified:   logs/192.168.99.7-telnet
        modified:   logs/192.168.99.8-telnet
        modified:   logs/192.168.99.9-telnet
        modified:   logs/192.168.99.92-telnet
        modified:   pid

no changes added to commit (use "git add" and/or "git commit -a")
root@nms01:/home/oxidized/.config/oxidized# git push
Username for 'https://git01.windows.domain.com': nms
Password for 'https://nms@git01.windows.domain.com':
Everything up-to-date

Let me know if more info is needed. Taking off for today also. will be back on tomorrow for this. Thanks

laf commented 6 years ago

repo: "/home/oxidized/.config/oxidixed/.git"

.git is a folder created within a repo so this seems like you've got the paths wrong.

/home/oxidized/.config/oxidized/ is the dir used when starting with the oxidized user for general stuff as you can see from the git status, point your the repo: config value to a new unused dir and try it again

vinesman commented 6 years ago

I have tried that, and get the same error.

New Config

output:
  default: git
  git:
    user: nms
    email: noreply1@domain.com
    repo: "/home/oxidized/.config/oxidixed/configs"

Error

Feb 23 10:20:00 nms01 oxidized[50540]: GithubRepo: Pushing local repository(/home/oxidized/.config/oxidixed/configs/)...
Feb 23 10:20:00 nms01 oxidized[50540]: GithubRepo: to remote: https://git01.windows.domain.com/Monitoring/networkconfigs.git
Feb 23 10:20:00 nms01 oxidized[50540]: GithubRepo: Using https auth
Feb 23 10:20:34 nms01 oxidized[50540]: GithubRepo: {:total_objects=>30, :indexed_objects=>30, :received_objects=>30, :local_objects=>0, :total_deltas=>10, :indexed_deltas=>10, :received_bytes=>0}
Feb 23 10:20:34 nms01 oxidized[50540]: GithubRepo: merging fetched branch origin/master
Feb 23 10:20:34 nms01 oxidized[50540]: Hook push_to_remote (#<GithubRepo:0x000000013e7640>) failed (#<TypeError: wrong argument type Fixnum (expected String)>) for event :post_store
Feb 23 10:20:34 nms01 oxidized[50540]: lib/oxidized/worker.rb: Jobs 0, Want: 1

It created the new objects and deltas, but still has the "wrong argument" in the hook push to remote.

Below are versions from gems installed


*** LOCAL GEMS ***

asetus (0.3.0)
backports (3.10.3)
bigdecimal (1.2.8)
did_you_mean (1.0.0)
emk-sinatra-url-for (0.2.1)
ffi (1.9.18)
haml (4.0.7)
htmlentities (4.3.4)
io-console (0.4.5)
json (1.8.3)
minitest (5.8.4)
multi_json (1.12.2)
mustermann (1.0.2)
net-ssh (3.0.2)
net-telnet (0.1.1)
oxidized (0.20.0)
oxidized-script (0.4.0)
oxidized-web (0.9.2)
power_assert (0.2.7)
psych (2.0.17)
puma (3.10.0)
rack (2.0.4, 1.6.8)
rack-protection (2.0.1, 1.5.3)
rack-test (0.7.0)
rake (10.5.0)
rb-fsevent (0.10.2)
rb-inotify (0.9.10)
rdoc (4.2.1)
rugged (0.26.0)
sass (3.5.2)
sass-listen (4.0.0)
sinatra (2.0.1, 1.4.8)
sinatra-contrib (1.4.7)
slop (3.6.0)
test-unit (3.1.7)

Git Version

git version 2.7.4

Linux Flavor

Linux nms01 4.4.0-97-generic #120-Ubuntu SMP Tue Sep 19 17:28:18 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

GitLab Version

GitLab 10.3.3
GitLab Shell 5.10.2
GitLab Workhorse v3.3.1
GitLab API v4
Gitaly 0.59.2
Git 2.14.3
Ruby 2.3.5p376
Rails 4.2.10
postgresql 9.6.5

Anything else ??

laf commented 6 years ago

I'm not sure what it could be now. Hopefully someone else may be able to offer some help

vinesman commented 6 years ago

Has no one else had this issue? Googling around, I see some similar, but not the same.

Could there be something with the git.rb file that is requiring something?

I would be more that happy to do a screen share of sorts to get this resolved.

vinesman commented 6 years ago

Here is what the git.rb looks like. I don't see any issues, but is there something in here that could cause the issue?

cat /var/lib/gems/2.3.0/gems/oxidized-0.20.0/lib/oxidized/output/git.rb
module Oxidized
class Git < Output
  class GitError < OxidizedError; end
  begin
    require 'rugged'
  rescue LoadError
    raise OxidizedError, 'rugged not found: sudo gem install rugged'
  end

  attr_reader :commitref

  def initialize
    @cfg = Oxidized.config.output.git
  end

  def setup
    if @cfg.empty?
      Oxidized.asetus.user.output.git.user  = 'Oxidized'
      Oxidized.asetus.user.output.git.email = 'o@example.com'
      Oxidized.asetus.user.output.git.repo  =  File.join(Config::Root, 'oxidized.git')
      Oxidized.asetus.save :user
      raise NoConfig, 'no output git config, edit ~/.config/oxidized/config'
    end

    if @cfg.repo.respond_to?(:each)
      @cfg.repo.each do |group, repo|
        @cfg.repo["#{group}="] = File.expand_path repo
      end
    else
      @cfg.repo = File.expand_path @cfg.repo
    end
  end

  def store file, outputs, opt={}
    @msg   = opt[:msg]
    @user  = (opt[:user]  or @cfg.user)
    @email = (opt[:email] or @cfg.email)
    @opt   = opt
    @commitref = nil
    repo   = @cfg.repo

    outputs.types.each do |type|
      type_cfg = ''
      type_repo = File.join(File.dirname(repo), type + '.git')
      outputs.type(type).each do |output|
        (type_cfg << output; next) if not output.name
        type_file = file + '--' + output.name
        if @cfg.type_as_directory?
          type_file = type + '/' + type_file
          type_repo = repo
        end
        update type_repo, type_file, output
      end
      update type_repo, file, type_cfg
    end

    update repo, file, outputs.to_cfg
  end

  def fetch node, group
    begin
      repo, path = yield_repo_and_path(node, group)
      repo = Rugged::Repository.new repo
      index = repo.index
      index.read_tree repo.head.target.tree unless repo.empty?
      repo.read(index.get(path)[:oid]).data
    rescue
      'node not found'
    end
  end

    # give a hash of all oid revision for the given node, and the date of the commit
    def version node, group
      begin
        repo, path = yield_repo_and_path(node, group)

        repo = Rugged::Repository.new repo
        walker = Rugged::Walker.new(repo)
        walker.sorting(Rugged::SORT_DATE)
        walker.push(repo.head.target)
        i = -1
        tab  = []
        walker.each do |commit|
          if commit.diff(paths: [path]).size > 0
            hash = {}
            hash[:date] = commit.time.to_s
            hash[:oid] = commit.oid
            hash[:author] = commit.author
            hash[:message] = commit.message
            tab[i += 1] = hash
          end
        end
        walker.reset
        tab
      rescue
        'node not found'
      end
    end

    #give the blob of a specific revision
    def get_version node, group, oid
      begin
        repo, path = yield_repo_and_path(node, group)
        repo = Rugged::Repository.new repo
        repo.blob_at(oid,path).content
      rescue
        'version not found'
      end
    end

    #give a hash with the patch of a diff between 2 revision and the stats (added and deleted lines)
    def get_diff node, group, oid1, oid2
      begin
        diff_commits = nil
        repo, _ = yield_repo_and_path(node, group)
        repo = Rugged::Repository.new repo
        commit = repo.lookup(oid1)

        if oid2
          commit_old = repo.lookup(oid2)
          diff = repo.diff(commit_old, commit)
          diff.each do |patch|
            if /#{node.name}\s+/.match(patch.to_s.lines.first)
              diff_commits = {:patch => patch.to_s, :stat => patch.stat}
              break
            end
          end
        else
          stat = commit.parents[0].diff(commit).stat
          stat = [stat[1],stat[2]]
          patch = commit.parents[0].diff(commit).patch
          diff_commits = {:patch => patch, :stat => stat}
        end

        diff_commits
      rescue
        'no diffs'
      end
    end

  private

  def yield_repo_and_path(node, group)
    repo, path = node.repo, node.name

    if group and @cfg.single_repo?
      path = "#{group}/#{node.name}"
    end

    [repo, path]
  end

  def update repo, file, data
    return if data.empty?

    if @opt[:group]
      if @cfg.single_repo?
        file = File.join @opt[:group], file
      else
        repo = if repo.is_a?(::String)
                 File.join File.dirname(repo), @opt[:group] + '.git'
               else
                 repo[@opt[:group]]
               end
      end
    end

    begin
      repo = Rugged::Repository.new repo
      update_repo repo, file, data, @msg, @user, @email
    rescue Rugged::OSError, Rugged::RepositoryError => open_error
      begin
        Rugged::Repository.init_at repo, :bare
      rescue => create_error
        raise GitError, "first '#{open_error.message}' was raised while opening git repo, then '#{create_error.message}' was while trying to create git repo"
      end
      retry
    end
  end

  def update_repo repo, file, data, msg, user, email
    oid = repo.write data, :blob
    index = repo.index
    index.read_tree repo.head.target.tree unless repo.empty?

    tree_old = index.write_tree repo
    index.add :path=>file, :oid=>oid, :mode=>0100644
    tree_new = index.write_tree repo

    if tree_old != tree_new
      repo.config['user.name']  = user
      repo.config['user.email'] = email
      @commitref = Rugged::Commit.create(repo,
        :tree       => index.write_tree(repo),
        :message    => msg,
        :parents    => repo.empty? ? [] : [repo.head.target].compact,
        :update_ref => 'HEAD',
      )

      index.write
      true
    end
  end
end
end
yzguy commented 6 years ago

Hello,

In /var/lib/gems/2.3.0/gems/oxidized-0.20.0/lib/oxidized/hook.rb ( https://github.com/ytti/oxidized/blob/master/lib/oxidized/hook.rb#L57-L63), can you set this block to be like

    @registered_hooks[event].each do |r_hook|
      #begin
        r_hook.hook.run_hook ctx
      #rescue => e
      #  Oxidized.logger.error "Hook #{r_hook.name} (#{r_hook.hook}) failed " +
      #           "(#{e.inspect}) for event #{event.inspect}"
      #end
    end

After commenting those lines out, it should not catch the exception when the hook is run and crashes, but it will give us the error trace, which will show what line it threw the exception on.

This line would give us a place to look in the Githubrepo hook to see what it's doing that it gets TypeError: wrong argument type Fixnum (expected String)

vinesman commented 6 years ago

I ended up rebuilding the host and everything works fine now. Not sure what the real cause was.

Ryan Vines Systems & Network Engineer Outdoor Adventurist 720 261 1405 ryan@abccolorado.com monster.vines@gmail.com www.abccolorado.com www.facebook.com/outdooradventuretime

On Wed, Mar 14, 2018 at 10:18 PM, Adam Smith notifications@github.com wrote:

Hello,

In https://github.com/ytti/oxidized/blob/master/lib/ oxidized/hook.rb#L57-L63, can you set this block to be like

@registered_hooks[event].each do |r_hook|
  #begin
    r_hook.hook.run_hook ctx
  #rescue => e
  #  Oxidized.logger.error "Hook #{r_hook.name} (#{r_hook.hook}) failed " +
  #           "(#{e.inspect}) for event #{event.inspect}"
  #end
end

After commenting those lines out, it should not catch the exception when the hook is run and crashes, but it will give us the error trace, which will show what line it threw the exception on.

This line would give us a place to look to see what it's doing that it gets TypeError: wrong argument type Fixnum (expected String)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ytti/oxidized/issues/1193#issuecomment-373255930, or mute the thread https://github.com/notifications/unsubscribe-auth/AGaQgzohH86UG-VOBlvy5UYRKCQclqN5ks5teeufgaJpZM4SQKf5 .

laf commented 6 years ago

Thanks for confirming. I'll close this down.