search

ytti / oxidized

Oxidized is a network device configuration backup tool. It's a RANCID replacement!
Apache License 2.0
2.8k stars 925 forks source link

Oxidized Fails to Start, Was Working Fine Earlier??? #3237

Closed MLyszyk closed 2 months ago

MLyszyk commented 3 months ago

@ytti Hi everyone, I am having some Oxidized issues if anyone can take a look please -->

sudo systemctl status oxidized
● oxidized.service - Oxidized - Network Device Configuration Backup Tool
     Loaded: loaded (/etc/systemd/system/oxidized.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Fri 2024-08-02 19:11:24 UTC; 2min 7s ago
    Process: 41569 ExecStart=/usr/local/bin/oxidized (code=exited, status=1/FAILURE)
   Main PID: 41569 (code=exited, status=1/FAILURE)

Aug 02 19:11:24 <device>.fakedomain.com systemd[1]: oxidized.service: Main process exited, code=exited, status=1/FAILURE
Aug 02 19:11:24 <device>.fakedomain.com systemd[1]: oxidized.service: Failed with result 'exit-code'.
sudo journalctl -fu oxidized
-- Logs begin at Fri 2024-05-31 12:50:59 UTC. --
Aug 02 19:06:48 <device>.fakedomain.com systemd[1]: Started Oxidized - Network Device Configuration Backup Tool.
Aug 02 19:06:48 <device>.fakedomain.com systemd[41297]: oxidized.service: Failed to execute command: No such file or directory
Aug 02 19:06:48 <device>.fakedomain.com systemd[41297]: oxidized.service: Failed at step EXEC spawning /home/mlyszyk/.rvm/gems/ruby-3.2.2/bin/oxidized: No such file or directory
Aug 02 19:06:48 <device>.fakedomain.com systemd[1]: oxidized.service: Main process exited, code=exited, status=203/EXEC
Aug 02 19:06:48 <device>.fakedomain.com systemd[1]: oxidized.service: Failed with result 'exit-code'.
Aug 02 19:11:14 <device>.fakedomain.com systemd[1]: Stopped Oxidized - Network Device Configuration Backup Tool.
Aug 02 19:11:14 <device>.fakedomain.com systemd[1]: Started Oxidized - Network Device Configuration Backup Tool.
Aug 02 19:11:24<device>.fakedomain.com oxidized[41569]: source returns no usable nodes
Aug 02 19:11:24 <device>.fakedomain.com systemd[1]: oxidized.service: Main process exited, code=exited, status=1/FAILURE
Aug 02 19:11:24 <device>.fakedomain.com systemd[1]: oxidized.service: Failed with result 'exit-code'.

In /home/oxidized/.config/oxidized/config -->

#------Source(s) for Oxidized to gather data from, no need to re-configure-----
source:
  default: csv
  csv:
    file: "/home/oxidized/.config/oxidized/CE_Devices.db"
    delimiter: !ruby/regexp /:/
    map:
      name: 0
      model: 1
      group: 2
      #username: 3
      #password: 4
    gpg: false

But the file is present and configured correctly -->

/home/oxidized/.config/oxidized$ ls -la
total 792
drwxrwxr-x 11 oxidized oxidized   4096 Aug  2 19:15 .
drwxrwxr-x  3 oxidized oxidized   4096 Aug 15  2023 ..
-rw-rw-r--  1 oxidized oxidized   3144 Jul 30 21:34 CE_Devices.db

If I try to run oxidized while I'm on my admin account, I get this -->

/home/oxidized/.config/oxidized$ oxidized
Traceback (most recent call last):
        2: from /usr/local/bin/oxidized:23:in `<main>'
        1: from /usr/lib/ruby/2.7.0/rubygems.rb:294:in `activate_bin_path'
/usr/lib/ruby/2.7.0/rubygems.rb:275:in `find_spec_for_exe': can't find gem oxidized (>= 0.a) with executable oxidized (Gem::GemNotFoundException)

However, If I run oxidized while being the oxidized user, I get this -->

:~/.config/oxidized$ oxidized
source returns no usable nodes

Here is the contents of the /usr/local/bin/oxidized file -->

#!/usr/bin/ruby2.7
#
# This file was generated by RubyGems.
#
# The application 'oxidized' is installed as part of a gem, and
# this file is here to facilitate running it.
#

require 'rubygems'

version = ">= 0.a"

str = ARGV.first
if str
  str = str.b[/\A_(.*)_\z/, 1]
  if str and Gem::Version.correct?(str)
    version = str
    ARGV.shift
  end
end

if Gem.respond_to?(:activate_bin_path)
load Gem.activate_bin_path('oxidized', 'oxidized', version)
else
gem "oxidized", version
load Gem.bin_path("oxidized", "oxidized", version)
end

Here is the content of the /lib/systemd/system/oxidized.service file -->

# Put this file in /etc/systemd/system.
#
# To set OXIDIZED_HOME instead of the default,
# ~oxidized/.config/oxidized, uncomment (and modify as required) the
# "Environment" variable below so systemd sets the correct
# environment.

[Unit]
Description=Oxidized - Network Device Configuration Backup Tool
After=network-online.target multi-user.target
Wants=network-online.target

[Service]
ExecStart=/usr/local/bin/oxidized
User=oxidized
KillSignal=SIGKILL
#Environment="OXIDIZED_HOME=/etc/oxidized"
Restart=on-failure
RestartSec=300s

[Install]
WantedBy=multi-user.target

Here is the content of the /etc/systemd/system/oxidized.service file -->

# Put this file in /etc/systemd/system.
#
# To set OXIDIZED_HOME instead of the default,
# ~oxidized/.config/oxidized, uncomment (and modify as required) the
# "Environment" variable below so systemd sets the correct
# environment.

[Unit]
Description=Oxidized - Network Device Configuration Backup Tool
After=network-online.target multi-user.target
Wants=network-online.target

[Service]
ExecStart=/usr/local/bin/oxidized
User=oxidized
KillSignal=SIGKILL
#Environment="OXIDIZED_HOME=/etc/oxidized"
Restart=on-failure
RestartSec=300s

[Install]
WantedBy=multi-user.target

If I do a "gem list | grep oxidized" command, this is what i see -->

/etc/systemd/system$ gem list | grep oxidized
oxidized (0.28.0, 0.26.3)
oxidized-script (0.6.0)
oxidized-web (0.13.1)

If I do a "which ruby" command and a "ruby --version" command from both my admin account and the oxidized account, this is what I see -->

oxidized@<device>:/etc/systemd/system$ which ruby
/usr/bin/ruby
oxidized@<device>:/etc/systemd/system$ ruby --version
ruby 2.7.0p0 (2019-12-25 revision 647ee6f091) [x86_64-linux-gnu]
oxidized@<device>:/etc/systemd/system$ exit
exit
mlyszyk@<device>:~$ which ruby
/home/mlyszyk/.rvm/rubies/ruby-3.2.2/bin/ruby
mlyszyk@<device>:~$ ruby --version
ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux]

I check the Ruby version and gems -->

oxidized@<machine>:~/.config/oxidized$ gem environment
RubyGems Environment:
  - RUBYGEMS VERSION: 3.1.2
  - RUBY VERSION: 2.7.0 (2019-12-25 patchlevel 0) [x86_64-linux-gnu]
  - INSTALLATION DIRECTORY: /var/lib/gems/2.7.0
  - USER INSTALLATION DIRECTORY: /home/oxidized/.gem/ruby/2.7.0
  - RUBY EXECUTABLE: /usr/bin/ruby2.7
  - GIT EXECUTABLE: /usr/bin/git
  - EXECUTABLE DIRECTORY: /usr/local/bin
  - SPEC CACHE DIRECTORY: /home/oxidized/.gem/specs
  - SYSTEM CONFIGURATION DIRECTORY: /etc
  - RUBYGEMS PLATFORMS:
    - ruby
    - x86_64-linux
  - GEM PATHS:
     - /var/lib/gems/2.7.0
     - /home/oxidized/.gem/ruby/2.7.0
     - /usr/lib/ruby/gems/2.7.0
     - /usr/share/rubygems-integration/2.7.0
     - /usr/share/rubygems-integration/all
     - /usr/lib/x86_64-linux-gnu/rubygems-integration/2.7.0
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :backtrace => false
     - :bulk_threshold => 1000
  - REMOTE SOURCES:
     - https://rubygems.org/
  - SHELL PATH:
     - /usr/local/sbin
     - /usr/local/bin
     - /usr/sbin
     - /usr/bin
     - /sbin
     - /bin
     - /usr/games
     - /usr/local/games
     - /snap/bin

~/.config/oxidized$ gem which oxidized
/var/lib/gems/2.7.0/gems/oxidized-0.28.0/lib/oxidized.rb

I do a "gem env" command from both my admin account and the oxidized account and this is what I see -->

mlyszyk@<device>:~$ gem env
RubyGems Environment:
  - RUBYGEMS VERSION: 3.4.19
  - RUBY VERSION: 3.2.2 (2023-03-30 patchlevel 53) [x86_64-linux]
  - INSTALLATION DIRECTORY: /home/mlyszyk/.rvm/gems/ruby-3.2.2
  - USER INSTALLATION DIRECTORY: /home/mlyszyk/.gem/ruby/3.2.0
  - RUBY EXECUTABLE: /home/mlyszyk/.rvm/rubies/ruby-3.2.2/bin/ruby
  - GIT EXECUTABLE: /usr/bin/git
  - EXECUTABLE DIRECTORY: /home/mlyszyk/.rvm/gems/ruby-3.2.2/bin
  - SPEC CACHE DIRECTORY: /home/mlyszyk/.gem/specs
  - SYSTEM CONFIGURATION DIRECTORY: /etc
  - RUBYGEMS PLATFORMS:
     - ruby
     - x86_64-linux
  - GEM PATHS:
     - /home/mlyszyk/.rvm/gems/ruby-3.2.2
     - /home/mlyszyk/.rvm/rubies/ruby-3.2.2/lib/ruby/gems/3.2.0
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :backtrace => true
     - :bulk_threshold => 1000
  - REMOTE SOURCES:
     - https://rubygems.org/
  - SHELL PATH:
     - /home/mlyszyk/.rvm/gems/ruby-3.2.2/bin
     - /home/mlyszyk/.rvm/gems/ruby-3.2.2@global/bin
     - /home/mlyszyk/.rvm/rubies/ruby-3.2.2/bin
     - /usr/local/sbin
     - /usr/local/bin
     - /usr/sbin
     - /usr/bin
     - /sbin
     - /bin
     - /usr/games
     - /usr/local/games
     - /snap/bin
     - /home/mlyszyk/.rvm/bin
     - /home/mlyszyk/.rvm/bin
mlyszyk@<device>:~$ sudo su oxidized
oxidized@<device>:/home/mlyszyk$ gem env
RubyGems Environment:
  - RUBYGEMS VERSION: 3.1.2
  - RUBY VERSION: 2.7.0 (2019-12-25 patchlevel 0) [x86_64-linux-gnu]
  - INSTALLATION DIRECTORY: /var/lib/gems/2.7.0
  - USER INSTALLATION DIRECTORY: /home/oxidized/.gem/ruby/2.7.0
  - RUBY EXECUTABLE: /usr/bin/ruby2.7
  - GIT EXECUTABLE: /usr/bin/git
  - EXECUTABLE DIRECTORY: /usr/local/bin
  - SPEC CACHE DIRECTORY: /home/oxidized/.gem/specs
  - SYSTEM CONFIGURATION DIRECTORY: /etc
  - RUBYGEMS PLATFORMS:
    - ruby
    - x86_64-linux
  - GEM PATHS:
     - /var/lib/gems/2.7.0
     - /home/oxidized/.gem/ruby/2.7.0
     - /usr/lib/ruby/gems/2.7.0
     - /usr/share/rubygems-integration/2.7.0
     - /usr/share/rubygems-integration/all
     - /usr/lib/x86_64-linux-gnu/rubygems-integration/2.7.0
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :backtrace => false
     - :bulk_threshold => 1000
  - REMOTE SOURCES:
     - https://rubygems.org/
  - SHELL PATH:
     - /usr/local/sbin
     - /usr/local/bin
     - /usr/sbin
     - /usr/bin
     - /sbin
     - /bin
     - /usr/games
     - /usr/local/games
     - /snap/bin

I tried to run the "sudo gem install oxidized" command from the oxidized account, and I got this -->

oxidized@<device>:/home/mlyszyk$ sudo gem install oxidized
Fetching net-scp-4.0.0.gem
Fetching net-protocol-0.2.2.gem
Fetching net-ftp-0.3.7.gem
Fetching time-0.3.0.gem
Fetching oxidized-0.30.1.gem
Successfully installed net-scp-4.0.0
Successfully installed time-0.3.0
Successfully installed net-protocol-0.2.2
Successfully installed net-ftp-0.3.7
ERROR:  Error installing oxidized:
        The last version of oxidized (>= 0) to support your Ruby & RubyGems was 0.28.0. Try installing it with `gem install oxidized -v 0.28.0`
        oxidized requires Ruby version >= 3.0. The current ruby version is 2.7.0.0.
mlyszyk@<device>:~$ which oxidized
/usr/local/bin/oxidized
mlyszyk@<device>:~$ sudo su oxidized
oxidized@<device>:/home/mlyszyk$ which oxidized
/usr/local/bin/oxidized

Gem info and Oxidized Version:

oxidized@<device>:~$ oxidized --version
0.28.0
oxidized@<device>:~$ gem info oxidized

*** LOCAL GEMS ***

oxidized (0.28.0, 0.26.3)
    Authors: Saku Ytti, Samer Abdel-Hafez, Anton Aksola
    Homepage: http://github.com/ytti/oxidized
    License: Apache-2.0
    Installed at (0.28.0): /var/lib/gems/2.7.0
                 (0.26.3): /var/lib/gems/2.7.0

    feeble attempt at rancid
oxidized@<device>:~$ exit
logout
mlyszyk@<device>:~$ gem info oxidized

*** LOCAL GEMS ***

mlyszyk@<device>:~$ oxidized --version
Traceback (most recent call last):
        2: from /usr/local/bin/oxidized:23:in `<main>'
        1: from /usr/lib/ruby/2.7.0/rubygems.rb:294:in `activate_bin_path'
/usr/lib/ruby/2.7.0/rubygems.rb:275:in `find_spec_for_exe': can't find gem oxidized (>= 0.a) with executable oxidized (Gem::GemNotFoundException)
mlyszyk@<device>:~$ gem list | grep oxidized
mlyszyk@<device>:~$ sudo gem list | grep oxidized
oxidized (0.28.0, 0.26.3)
oxidized-script (0.6.0)
oxidized-web (0.13.1)
mlyszyk@<device>:~$ sudo su oxidized
oxidized@<device>:/home/mlyszyk$ gem list | grep oxidized
oxidized (0.28.0, 0.26.3)
oxidized-script (0.6.0)
oxidized-web (0.13.1)

Time and Scenario when issue occurred: I was trying to see if I can adjust how Oxidized formats its emails that get sent out. The only file I modified was "/opt/oxidized/oxidized-report-git-commits". I was trying to do some formatting changes, but I reverted the file back to how it was, primarily this line -->

git --bare --git-dir="${OX_REPO_NAME}" show --pretty='' --no-color "${OX_REPO_COMMITREF}"

Now oxidized is being weird and won't start, even though it was working just fine earlier today. I know I added a lot of info bit wanted to add as much as I could to help streamline troubleshooting compared to having people ask for this file and that file and such. If anything else is needed, please let me know! I think I have an idea of what is going on, but want to verify what's happening before I touch anything else since Oxidized seems to be so fragile.

MLyszyk commented 3 months ago

I also just updated the "/etc/systemd/system/oxidized.service" and "lib/systemd/system/oxidized.service" files and changed the #Environment="OXIDIZED_HOME=/etc/oxidized" to be Environment="OXIDIZED_HOME=/home/oxidized/.config/oxidized", but still won't start.

MLyszyk commented 3 months ago

One more thing, tried doing some ruby stuff and installing oxidized gems and this is where I am left off -->

oxidized@<device>/home/mlyszyk$ rvm list

Command 'rvm' not found, but there are 19 similar ones.

oxidized@<device>:/home/mlyszyk$ exit
exit
mlyszyk@<device>:~$ rvm use ruby-3.2.2
Using /home/mlyszyk/.rvm/gems/ruby-3.2.2
mlyszyk@<device>:~$ sudo su - oxidized
oxidized@<device>:~$ sudo gem install oxidized
ERROR:  Error installing oxidized:
        The last version of oxidized (>= 0) to support your Ruby & RubyGems was 0.28.0. Try installing it with `gem install oxidized -v 0.28.0`
        oxidized requires Ruby version >= 3.0. The current ruby version is 2.7.0.0.
oxidized@<device>:~$ sudo gem install oxidized -v 0.28.0
Successfully installed oxidized-0.28.0
Parsing documentation for oxidized-0.28.0
Done installing documentation for oxidized after 1 seconds
1 gem installed
oxidized@<device>:~$ sudo systemctl daemon-reload
oxidized@<device>:~$ sudo systemctl restart oxidized
oxidized@<device>:~$ sudo systemctl status oxidized
● oxidized.service - Oxidized - Network Device Configuration Backup Tool
     Loaded: loaded (/etc/systemd/system/oxidized.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Fri 2024-08-02 20:21:56 UTC; 1s ago
    Process: 44577 ExecStart=/usr/local/bin/oxidized (code=exited, status=1/FAILURE)
   Main PID: 44577 (code=exited, status=1/FAILURE)
oxidized@<device>:~$ sudo journalctl -fu oxidized
-- Logs begin at Fri 2024-05-31 12:50:59 UTC. --
Aug 02 20:21:22 <device>.fakedomain.com systemd[1]: Stopped Oxidized - Network Device Configuration Backup Tool.
Aug 02 20:21:22 <device>.fakedomain.com systemd[1]: Started Oxidized - Network Device Configuration Backup Tool.
Aug 02 20:21:26 <device>.fakedomain.com oxidized[44496]: source returns no usable nodes
Aug 02 20:21:26 <device>.fakedomain.com systemd[1]: oxidized.service: Main process exited, code=exited, status=1/FAILURE
Aug 02 20:21:26 <device>.fakedomain.com systemd[1]: oxidized.service: Failed with result 'exit-code'.
Aug 02 20:21:53 <device>.fakedomain.com systemd[1]: Stopped Oxidized - Network Device Configuration Backup Tool.
Aug 02 20:21:53 <device>.fakedomain.com systemd[1]: Started Oxidized - Network Device Configuration Backup Tool.
Aug 02 20:21:56 <device>.fakedomain.com oxidized[44577]: source returns no usable nodes
Aug 02 20:21:56<device>.fakedomain.com systemd[1]: oxidized.service: Main process exited, code=exited, status=1/FAILURE
Aug 02 20:21:56 <device>.fakedomain.com systemd[1]: oxidized.service: Failed with result 'exit-code'.
robertcheramy commented 2 months ago

It is written in the first lines:

Aug 02 19:06:48 <device>.fakedomain.com systemd[41297]: oxidized.service: Failed to execute command: No such file or directory
Aug 02 19:06:48 <device>.fakedomain.com systemd[41297]: oxidized.service: Failed at step EXEC spawning /home/mlyszyk/.rvm/gems/ruby-3.2.2/bin/oxidized: No such file or directory

You probably updated ruby, and the path ist another.

This is not an oxidized issue, closing.