search

ytti / oxidized

Oxidized is a network device configuration backup tool. It's a RANCID replacement!
Apache License 2.0
2.8k stars 925 forks source link

Some routeros devices not working #3032

Closed tonoitp closed 9 months ago

tonoitp commented 10 months ago

Hi, Some -but not all- of my mikrotik devices don't backup. In the router I get an error "auth timeout" so I tested from the commandline. And indeed, ssh admin@192.168.0.1 left me with no reaction. Then I found ssh -o MACs=hmac-sha2-256 admin@192.168.0.1 to be working.

So I added a little to the config

input:
  default: ssh
  ssh:
    secure: false
    ssh_hmac: hmac-sha2-256

but no difference. Tried the same under source / vars_map and the hmac-sha2-256 value in router.db but no success there either. The logs made me nothing wiser, and SSH logfiles are zero bytes for ssh is connected, but no data send/received yet.

Any suggestions?


I, [2024-01-12T13:18:18.472505 #2405]  INFO -- : Oxidized starting, running as pid 2405
I, [2024-01-12T13:18:18.473205 #2405]  INFO -- : lib/oxidized/nodes.rb: Loading nodes
D, [2024-01-12T13:18:18.473341 #2405] DEBUG -- : resolving DNS for mikrotik-rb4011...
D, [2024-01-12T13:18:18.473376 #2405] DEBUG -- : IPADDR 192.168.0.1
D, [2024-01-12T13:18:18.473459 #2405] DEBUG -- : node.rb: resolving node key 'model', with passed global value of '' and node value 'routeros'
D, [2024-01-12T13:18:18.473573 #2405] DEBUG -- : node.rb: setting node key 'model' to value 'junos' from global
D, [2024-01-12T13:18:18.473626 #2405] DEBUG -- : node.rb: returning node key 'model' with value 'routeros'
D, [2024-01-12T13:18:18.473661 #2405] DEBUG -- : lib/oxidized/node.rb: Loading model "routeros"
D, [2024-01-12T13:18:18.475693 #2405] DEBUG -- : lib/oxidized/model/model.rb Added all to the commands list
D, [2024-01-12T13:18:18.475785 #2405] DEBUG -- : lib/oxidized/model/model.rb Added /system routerboard print without-paging to the commands list
D, [2024-01-12T13:18:18.475836 #2405] DEBUG -- : lib/oxidized/model/model.rb Added /system package update print without-paging to the commands list
D, [2024-01-12T13:18:18.475871 #2405] DEBUG -- : lib/oxidized/model/model.rb Added /system history print without-paging to the commands list
D, [2024-01-12T13:18:18.476391 #2405] DEBUG -- : node.rb: resolving node key 'input', with passed global value of 'ssh' and node value ''
D, [2024-01-12T13:18:18.476470 #2405] DEBUG -- : node.rb: returning node key 'input' with value 'ssh'
D, [2024-01-12T13:18:18.641387 #2405] DEBUG -- : node.rb: resolving node key 'output', with passed global value of 'file' and node value ''
D, [2024-01-12T13:18:18.641468 #2405] DEBUG -- : node.rb: returning node key 'output' with value 'file'
D, [2024-01-12T13:18:18.642011 #2405] DEBUG -- : node.rb: resolving node key 'username', with passed global value of '' and node value ''
D, [2024-01-12T13:18:18.642060 #2405] DEBUG -- : node.rb: setting node key 'username' to value 'admin' from global
D, [2024-01-12T13:18:18.642090 #2405] DEBUG -- : node.rb: returning node key 'username' with value 'admin'
D, [2024-01-12T13:18:18.642110 #2405] DEBUG -- : node.rb: resolving node key 'password', with passed global value of '' and node value ''
D, [2024-01-12T13:18:18.642132 #2405] DEBUG -- : node.rb: setting node key 'password' to value 'admin' from global
D, [2024-01-12T13:18:18.642154 #2405] DEBUG -- : node.rb: returning node key 'password' with value 'admin'
I, [2024-01-12T13:18:18.642213 #2405]  INFO -- : lib/oxidized/nodes.rb: Loaded 1 nodes
D, [2024-01-12T13:18:18.970206 #2405] DEBUG -- : lib/oxidized/core.rb: Starting the worker...
Puma starting in single mode...
* Version 3.11.4 (ruby 3.1.2-p20), codename: Love Song
* Min threads: 0, max threads: 16
* Environment: development
* Listening on tcp://127.0.0.1:8888
Use Ctrl-C to stop
D, [2024-01-12T13:18:19.971432 #2405] DEBUG -- : lib/oxidized/worker.rb: Jobs running: 0 of 1 - ended: 0 of 1
D, [2024-01-12T13:18:19.971986 #2405] DEBUG -- : lib/oxidized/worker.rb: Added /mikrotik-rb4011 to the job queue
D, [2024-01-12T13:18:19.972041 #2405] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2024-01-12T13:18:19.972253 #2405] DEBUG -- : lib/oxidized/job.rb: Starting fetching process for mikrotik-rb4011 at 2024-01-12 12:18:19 UTC
D, [2024-01-12T13:18:19.972639 #2405] DEBUG -- : lib/oxidized/input/ssh.rb: Connecting to mikrotik-rb4011
D, [2024-01-12T13:18:19.972838 #2405] DEBUG -- : AUTH METHODS::["none", "publickey", "password"]
D, [2024-01-12T13:18:19.973874 #2405] DEBUG -- net.ssh.transport.session[514]: establishing connection to 192.168.0.1:22
D, [2024-01-12T13:18:19.975326 #2405] DEBUG -- net.ssh.transport.session[514]: connection established
I, [2024-01-12T13:18:19.975532 #2405]  INFO -- net.ssh.transport.server_version[528]: negotiating protocol version
D, [2024-01-12T13:18:19.975683 #2405] DEBUG -- net.ssh.transport.server_version[528]: local is `SSH-2.0-Ruby/Net::SSH_7.2.1 x86_64-linux-gnu'
D, [2024-01-12T13:18:19.985266 #2405] DEBUG -- net.ssh.transport.server_version[528]: remote is `SSH-2.0-ROSSSH'
I, [2024-01-12T13:18:19.985696 #2405]  INFO -- net.ssh.transport.algorithms[53c]: sending KEXINIT
D, [2024-01-12T13:18:19.986006 #2405] DEBUG -- socket[550]: queueing packet nr 0 type 20 len 1436
D, [2024-01-12T13:18:19.986167 #2405] DEBUG -- socket[550]: sent 1440 bytes
D, [2024-01-12T13:18:19.986360 #2405] DEBUG -- socket[550]: read 232 bytes
D, [2024-01-12T13:18:19.986527 #2405] DEBUG -- socket[550]: received packet nr 0 type 20 len 228
I, [2024-01-12T13:18:19.986671 #2405]  INFO -- net.ssh.transport.algorithms[53c]: got KEXINIT from server
I, [2024-01-12T13:18:19.986844 #2405]  INFO -- net.ssh.transport.algorithms[53c]: negotiating algorithms
D, [2024-01-12T13:18:19.987041 #2405] DEBUG -- net.ssh.transport.algorithms[53c]: negotiated:
* kex: diffie-hellman-group-exchange-sha256
* host_key: ssh-rsa
* encryption_server: aes256-ctr
* encryption_client: aes256-ctr
* hmac_client: hmac-sha2-256
* hmac_server: hmac-sha2-256
* compression_client: none
* compression_server: none
* language_client:
* language_server:
D, [2024-01-12T13:18:19.987080 #2405] DEBUG -- net.ssh.transport.algorithms[53c]: exchanging keys
D, [2024-01-12T13:18:19.988492 #2405] DEBUG -- socket[550]: queueing packet nr 1 type 34 len 20
D, [2024-01-12T13:18:19.988716 #2405] DEBUG -- socket[550]: sent 24 bytes
D, [2024-01-12T13:18:20.973269 #2405] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2024-01-12T13:18:21.974513 #2405] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2024-01-12T13:18:22.975764 #2405] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2024-01-12T13:18:23.977019 #2405] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2024-01-12T13:18:24.978264 #2405] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2024-01-12T13:18:25.979518 #2405] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2024-01-12T13:18:26.980793 #2405] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2024-01-12T13:18:27.982050 #2405] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2024-01-12T13:18:28.983302 #2405] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2024-01-12T13:18:29.984632 #2405] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2024-01-12T13:18:30.985903 #2405] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2024-01-12T13:18:31.987129 #2405] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2024-01-12T13:18:32.988348 #2405] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2024-01-12T13:18:33.989596 #2405] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2024-01-12T13:18:34.990854 #2405] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2024-01-12T13:18:35.992116 #2405] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2024-01-12T13:18:36.993390 #2405] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2024-01-12T13:18:37.994618 #2405] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2024-01-12T13:18:38.995840 #2405] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2024-01-12T13:18:39.997058 #2405] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
W, [2024-01-12T13:18:40.009142 #2405]  WARN -- : 192.168.0.1 raised Net::SSH::ConnectionTimeout (rescued RuntimeError) with msg "timeout waiting for next packet"
D, [2024-01-12T13:18:40.009483 #2405] DEBUG -- : lib/oxidized/node.rb: Oxidized::SSH failed for mikrotik-rb4011
D, [2024-01-12T13:18:40.009898 #2405] DEBUG -- : lib/oxidized/job.rb: Config fetched for mikrotik-rb4011 at 2024-01-12 12:18:40 UTC
W, [2024-01-12T13:18:40.998819 #2405]  WARN -- : /mikrotik-rb4011 status no_connection, retry attempt 1
D, [2024-01-12T13:18:40.998939 #2405] DEBUG -- : lib/oxidized/worker.rb: Jobs running: 0 of 1 - ended: 0 of 1
D, [2024-01-12T13:18:40.999024 #2405] DEBUG -- : lib/oxidized/worker.rb: Added /mikrotik-rb4011 to the job queue
D, [2024-01-12T13:18:40.999066 #2405] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2024-01-12T13:18:40.999219 #2405] DEBUG -- : lib/oxidized/job.rb: Starting fetching process for mikrotik-rb4011 at 2024-01-12 12:18:40 UTC
D, [2024-01-12T13:18:40.999479 #2405] DEBUG -- : lib/oxidized/input/ssh.rb: Connecting to mikrotik-rb4011
D, [2024-01-12T13:18:40.999566 #2405] DEBUG -- : AUTH METHODS::["none", "publickey", "password"]
D, [2024-01-12T13:18:41.000313 #2405] DEBUG -- net.ssh.transport.session[58c]: establishing connection to 192.168.0.1:22
D, [2024-01-12T13:18:41.001692 #2405] DEBUG -- net.ssh.transport.session[58c]: connection established
I, [2024-01-12T13:18:41.001777 #2405]  INFO -- net.ssh.transport.server_version[5a0]: negotiating protocol version
D, [2024-01-12T13:18:41.001806 #2405] DEBUG -- net.ssh.transport.server_version[5a0]: local is `SSH-2.0-Ruby/Net::SSH_7.2.1 x86_64-linux-gnu'
D, [2024-01-12T13:18:41.002656 #2405] DEBUG -- net.ssh.transport.server_version[5a0]: remote is `SSH-2.0-ROSSSH'
I, [2024-01-12T13:18:41.002915 #2405]  INFO -- net.ssh.transport.algorithms[5b4]: sending KEXINIT
D, [2024-01-12T13:18:41.003067 #2405] DEBUG -- socket[5c8]: queueing packet nr 0 type 20 len 1436
D, [2024-01-12T13:18:41.003138 #2405] DEBUG -- socket[5c8]: sent 1440 bytes
D, [2024-01-12T13:18:41.003211 #2405] DEBUG -- socket[5c8]: read 232 bytes
D, [2024-01-12T13:18:41.003478 #2405] DEBUG -- socket[5c8]: received packet nr 0 type 20 len 228
I, [2024-01-12T13:18:41.003523 #2405]  INFO -- net.ssh.transport.algorithms[5b4]: got KEXINIT from server
I, [2024-01-12T13:18:41.003578 #2405]  INFO -- net.ssh.transport.algorithms[5b4]: negotiating algorithms
D, [2024-01-12T13:18:41.003667 #2405] DEBUG -- net.ssh.transport.algorithms[5b4]: negotiated:
* kex: diffie-hellman-group-exchange-sha256
* host_key: ssh-rsa
* encryption_server: aes256-ctr
* encryption_client: aes256-ctr
* hmac_client: hmac-sha2-256
* hmac_server: hmac-sha2-256
* compression_client: none
* compression_server: none
* language_client:
* language_server:
D, [2024-01-12T13:18:41.003706 #2405] DEBUG -- net.ssh.transport.algorithms[5b4]: exchanging keys
D, [2024-01-12T13:18:41.003833 #2405] DEBUG -- socket[5c8]: queueing packet nr 1 type 34 len 20
D, [2024-01-12T13:18:41.003869 #2405] DEBUG -- socket[5c8]: sent 24 bytes
D, [2024-01-12T13:18:42.000300 #2405] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
tonoitp commented 9 months ago

I created .ssh/config and added

host *
  MACs=hmac-sha2-256

now ssh works without specifying the option (obvious) but it made no difference for oxidized

ajsiersema commented 9 months ago

Instead of changing the parameters globally, try grouping the exceptions. This should work:

groups:
  routeros_hmac-sha2-256:
    vars:
      ssh_hmac: hmac-sha2-256
tonoitp commented 9 months ago

Thank you for the suggestion. It did not solve the issue :( I did more testing and found a bit more. I'll close this one and open a new one with an better description