search

ytti / oxidized

Oxidized is a network device configuration backup tool. It's a RANCID replacement!
Apache License 2.0
2.76k stars 917 forks source link

incomplete cisco config (no show running config output)on router and multi layer switch #1958

Closed kutsheax closed 1 year ago

kutsheax commented 4 years ago

Hello Oxidized Guru's,

I am having incomplete output on my cisco device on router and multilayer switch. below are the UI output.

I've tried adjusting with timeouts, create groups without any success.

Router UI Output:

! Cisco IOS Software, C3900 Software (C3900-UNIVERSALK9-M), Version 15.4(3)M3, RELEASE SOFTWARE (fc2) ! ! Image: Software: C3900-UNIVERSALK9-M, 15.4(3)M3, RELEASE SOFTWARE (fc2) ! Image: Compiled: Fri 05-Jun-15 15:47 by prod_rel_team ! ROM Bootstrap: Version 15.0(1r)M16, RELEASE SOFTWARE (fc1) ! Image: flash0:c3900-universalk9-mz.SPA.154-3.M3.bin ! Chassis type: CISCO3945-CHASSIS ! Memory: main 978944K/69632K ! Processor ID: XXXXXXXXXX ! CPU: ! Memory: nvram 255K ! ! VTP: VTP Version : 2 ! VTP: Configuration Revision : 1 ! VTP: Maximum VLANs supported locally : 68 ! VTP: Number of existing VLANs : 6 ! VTP: VTP Operating Mode : Server ! VTP: VTP Domain Name : ! VTP: VTP Pruning Mode : Disabled ! VTP: VTP V2 Mode : Disabled ! VTP: VTP Traps Generation : Disabled ! VTP: MD5 digest : 0xDF 0xD5 0x80 0x5C 0xF0 0x96 0x47 0xEB ! VTP: Configuration last modified by 112.199.103.234 at 2-1-18 03:52:31 ! VTP: Local updater ID is 111.111.111.111 on interface Gi0/0 (first interface found) !

Multi layer switch UI output: ! show version ! ! ! VTP: show vtp status !

CLI Version: Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500es8-UNIVERSALK9-M), Version 03.06.03.E RELEASE SOFTWARE (fc3) Technical Support: http://www.cisco.com/techsupport Copyright (c) 1986-2015 by Cisco Systems, Inc. Compiled Wed 26-Aug-15 20:41 by prod_rel_team

Cisco IOS-XE software, Copyright (c) 2005-2015 by cisco Systems, Inc. All rights reserved. Certain components of Cisco IOS-XE software are licensed under the GNU General Public License ("GPL") Version 2.0. The software code licensed under GPL Version 2.0 is free software that comes with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such GPL code under the terms of GPL Version 2.0. (http://www.gnu.org/licenses/gpl-2.0.html) For more details, see the documentation or "License Notice" file accompanying the IOS-XE software, or the applicable URL provided on the flyer accompanying the IOS-XE software.

ROM: 15.1(1r)SG5

License Information for 'WS-X45-SUP8-E' License Level: entservices Type: Permanent Next reboot license Level: entservices

cisco WS-C4510R+E (P5040) processor (revision 2) with 4194304K bytes of physical memory. Processor board ID XXXXXXX P5040 CPU at 2.2GHz, Supervisor 8-E

Debug:

Multilayer switch: Nov 4 18:43:01 phv-netad-config-server-01 oxidized: I, [2019-11-04T18:43:01.749297 #128404] INFO -- : Configuration updated for /me-dist Nov 4 18:43:01 phv-netad-config-server-01 oxidized: D, [2019-11-04T18:43:01.749390 #128404] DEBUG -- : lib/oxidized/worker.rb: Jobs running: 0 of 1 - ended: 1 of 3

on router: Nov 4 18:43:01 phv-netad-config-server-01 oxidized: D, [2019-11-04T18:43:01.749513 #128404] DEBUG -- : lib/oxidized/worker.rb: Added /me-edge1 to the job queue Nov 4 18:43:01 phv-netad-config-server-01 oxidized: D, [2019-11-04T18:43:01.749535 #128404] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel Nov 4 18:43:01 phv-netad-config-server-01 oxidized: D, [2019-11-04T18:43:01.749643 #128404] DEBUG -- : lib/oxidized/job.rb: Starting fetching process for me-edge1 at 2019-11-04 10:43:01 UTC Nov 4 18:43:01 phv-netad-config-server-01 oxidized: D, [2019-11-04T18:43:01.749835 #128404] DEBUG -- : lib/oxidized/input/ssh.rb: Connecting to me-edge1 Nov 4 18:43:01 phv-netad-config-server-01 oxidized: D, [2019-11-04T18:43:01.749937 #128404] DEBUG -- : AUTH METHODS::["none", "publickey", "password"] Nov 4 18:43:02 phv-netad-config-server-01 oxidized: D, [2019-11-04T18:43:02.749908 #128404] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel Nov 4 18:43:03 phv-netad-config-server-01 oxidized: D, [2019-11-04T18:43:03.750240 #128404] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel Nov 4 18:43:04 phv-netad-config-server-01 oxidized: D, [2019-11-04T18:43:04.750515 #128404] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel Nov 4 18:43:05 phv-netad-config-server-01 oxidized: D, [2019-11-04T18:43:05.750850 #128404] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel Nov 4 18:43:06 phv-netad-config-server-01 oxidized: D, [2019-11-04T18:43:06.751211 #128404] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel

oxidized version: [root@phv-netad-config-server-01 ~]# oxidized --version 0.26.3

CONFIG FILE: username: oxidized password: oxidized model: junos resolve_dns: true interval: 3600 use_syslog: false debug: true threads: 30 timeout: 60 retries: 3 prompt: !ruby/regexp /^([\w.@-]+[#>]\s?)$/ rest: 0.0.0.0:8888 next_adds_job: false vars: {} groups: default: username: oxidized password: oxidized vars: enable: oxidized ios: username: oxidized password: oxidized vars: enable: oxidized

models: {} pid: "/home/oxidized/.config/oxidized/pid" crash: directory: "/home/oxidized/.config/oxidized/crashes" hostnames: false

stats: history_size: 10 input: default: ssh, telnet debug: false ssh: secure: false ftp: passive: true utf8_encoded: true output: default: file file: directory: "/root/.config/oxidized/configs" source: default: csv csv: file: "/root/.config/oxidized/router.db" delimiter: !ruby/regexp /:/ map: name: 0 model: 1 gpg: false model_map: juniper: junos cisco: iosxe cisco1: ios

cisco2: iosxr

I want to resolve this issue before i add the librenms.

appreciate any help.

thanks,

maxim-kyryliuk commented 4 years ago

Maybe i have something similar #1966

brna62petsto commented 4 years ago

I also have a similar issue. Getting switch config (Arista) using EOS model (https://github.com/ytti/oxidized/blob/master/lib/oxidized/model/eos.rb) fails SOMETIMES - on average every second or third backup run. I couldn't catch the pattern apart that this only happens for switches with larger configs, greater then 380k. $ oxidized --version 0.27.0 @ytti any clue on how to approach this is appreciated

maxim-kyryliuk commented 4 years ago

@brna62petsto Hi, i think this is issue not in oxidized but in net/ssh. I tried use pure ruby script that will show ssh command output and get same issue (output truncated) i already submit issue in net/ssh https://github.com/net-ssh/net-ssh/issues/726 it will be good if you can also submit or comment it. It can get attention to this issue.

ytti commented 4 years ago

Great work @nataku, thank you!

ytti commented 4 years ago

Does this always work with OpenSSH? ssh router "command"

maxim-kyryliuk commented 4 years ago

I can't say that it work at 100% time coz i don't use SSH for backup. I am done this only for test. But i just made 10 runs of ruby script and OpenSSH and OpenSSH show result all time and ruby script all 10 times truncated output,

brna62petsto commented 4 years ago

@brna62petsto Hi, i think this is issue not in oxidized but in net/ssh. I tried use pure ruby script that will show ssh command output and get same issue (output truncated) i already submit issue in net/ssh net-ssh/net-ssh#726 it will be good if you can also submit or comment it. It can get attention to this issue.

Hi @nataku,

Thanks for the effort but I think we might have a slightly different issue. When I tail -f log file of the affected device (/var/lib/oxidized/config/oxidized/logs/-ssh) I always see the complete configuration. I believe that part is net/ssh. But, when the output goes through the oxidized processing it sometimes gets truncated (/var/lib/oxidized/config/oxidized/nodes/). @nataku @ytti let me know if this makes sense.

maxim-kyryliuk commented 4 years ago

Hi @brna62petsto, How you enable this option? I enabled DEBUG and got debug output but i i can't see anything from command output.

Can you guide me how you make it? I will check my system and compare.

brna62petsto commented 4 years ago

@nataku
In the /var/lib/oxidized/config/oxidized/config file you can also enable debug mode for "input". It should look like this: input: default: ssh debug: true

Restart oxidized service and in the folder /var/lib/oxidized/config/oxidized/logs you will see files populated with what actual commands returned via (I believe) ruby net/ssh. Files have format -ssh.

Let me know what you find.

maxim-kyryliuk commented 4 years ago

Look like it is not working for my setup

username: oxidized password: password model: junos interval: 3600 #interval in seconds log: ~/.config/oxidized/log threads: 5 timeout: 60 retries: 3 prompt: !ruby/regexp /^([\w.@-]+[#>]\s?)$/ debug: true crash: directory: ~/.config/oxidized/crashes hostnames: false vars: enable: S3cr3tx ssh_keys: "~/.ssh/id_rsa" groups: {} rest: 127.0.0.1:8888 pid: ~/.config/oxidized/oxidized.pid input: default: ssh debug: true ssh: secure: false output: default: git git: user: Oxidized email: oxidized@example.com repo: "~/.config/oxidized/oxidized.git" source: default: csv csv: file: "~/.config/oxidized/router.db" delimiter: !ruby/regexp /:/ map: name: 0 model: 1 ip: 2 vars_map: ssh_port: 3 model_map: cisco: ios juniper: junos

I have files ip-ssh, but all of them 0 byte size, but date when files changed always changed, For example if i run backup i can see that file changed date but it size still 0.

I will try why it happen and dig it. Thanks for guide.

jaakub commented 4 years ago

Hi All,

Oxidized version - 0.28.0.

We are experiencing identical issue. All command output is logged expect show running-config.

It executes the following commands in the following order, and doesn't go into enable/privileged mode prior to show running-config command. Any ideas how to fix that? @ytti 


device-1>terminal width 0
device-1>show version
device-1>show vtp status
device-1>show inventory
device-1>show running-config
show running-config
      ^
% Invalid input detected at '^' marker.

[user@oxidized ~]# /usr/local/bin/oxidized --version
0.28.0```
jaakub commented 4 years ago

Actually, I've now worked it out. My var map was misconfigured and it now authenticates first thing.

Before:

    map:
      name: name
      ip: ip
      model: model
      group: group
      username: username
      password: password
      enable: enable

After:

    map:
      name: name
      ip: ip
      model: model
      group: group
      username: username
      password: password
    vars_map:
      enable: enable

Cheers

mortzu commented 2 years ago

Can you please reformat your paste to make it readable? (https://docs.github.com/en/github/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#quoting-code)

tim427 commented 2 years ago

Also got the same issue with one Mikrotik. An Mikrotik with an larger config, always truncated arnoud the same line/character.