search

ytti / oxidized

Oxidized is a network device configuration backup tool. It's a RANCID replacement!
Apache License 2.0
2.81k stars 928 forks source link

fail to launch as a service #556

Closed JfryR closed 8 years ago

JfryR commented 8 years ago

Hi,

I have a problem.

Oxidized run fine when I launch it with usr/local/share/gems/gems/oxidized-0.13.1/bin/oxidized

But when I try to launch it with systemd (copy oxidized.script) it failed with error:

ep 05 10:51:16 localhost oxidized[31395]: /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- oxidized/cli (LoadError)
Sep 05 10:51:16 localhost oxidized[31395]: from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
Sep 05 10:51:16 localhost systemd[1]: oxidized.service: main process exited, code=exited, status=1/FAILURE
Sep 05 10:51:16 localhost systemd[1]: Unit oxidized.service entered failed state.
Sep 05 10:51:16 localhost systemd[1]: oxidized.service failed.

Do you have a documentation for install it as a service correctly ?

Regards,

ytti commented 8 years ago

You've probably installed it with rvm, which means it's in rvm's directory structure and your operating system has no idea of that. You need to add in your init script initialisation of the rvm, like you have in your own user's shell rc file.

JfryR commented 8 years ago

I installed it with gem install

It seems its because I try to launch it with oxidized user and he didn't have all the rights he needed ?

EDIT: when i try it with root user it works

ytti commented 8 years ago

Perhaps your umask is 022. gem (unfortunately) honours your umask, so files installed likely are not user readable.

JfryR commented 8 years ago

oh okay, I understand better now, I must change all of rights on oxidized files to run it as oxidized user.

Last question, evenif oxidized-web is installed, when I launch with default REST config, it can't start because of : root/.gem/ruby/gems/oxidized-0.16.3/lib/oxidized/core.rb:22:inrescue in initialize': oxidized-web not found`

but when I did a gem list I found that oxidized-web is available:


*** LOCAL GEMS ***
oxidized (0.16.3)
oxidized-script (0.3.1)
oxidized-web (0.7.1)
ytti commented 8 years ago

Or just reinstall it with correct umask? umask 022; gem install oxidized

JfryR commented 8 years ago

Yes it seems better!

And do you have any information for oxidized-web problem ?

Thank you for yours quick replies

ytti commented 8 years ago

Maybe reinstall oxidized-web with correct umask as well? umask 022; gem install oxidized-web

JfryR commented 8 years ago

But for now I just test with root user so umask 022 won't be a problem here I think I try to uninstall oxidized-web and re-install it and same problem,

With strace, in error message I can see that it search for puma_http11

ytti commented 8 years ago

Do you have full backtrace of the crash

JfryR commented 8 years ago

trace_oxdweb.txt

See attached document :)

ytti commented 8 years ago

I don't mean strace, I mean the backtrace you get when oxidized crashed. Should be in ~/.config/oxidzed/ directory.

JfryR commented 8 years ago 
# /usr/local/bin/oxidized -d                                                                                                                                                                            
I, [2016-09-05T12:33:59.327174 #51453]  INFO -- : Oxidized starting, running as pid 51453                                                                                                                                                    
I, [2016-09-05T12:33:59.327987 #51453]  INFO -- : lib/oxidized/nodes.rb: Loading nodes                                                                                                                                                       
D, [2016-09-05T12:33:59.328219 #51453] DEBUG -- : resolving DNS for sw2960-banc-test.infra.local...                                                                                                                                          
D, [2016-09-05T12:33:59.403503 #51453] DEBUG -- : lib/oxidized/node.rb: Loading model "ios"                                                                                                                                                  
D, [2016-09-05T12:33:59.405256 #51453] DEBUG -- : lib/oxidized/model/model.rb Added all to the commands list                                                                                                                                 
D, [2016-09-05T12:33:59.405346 #51453] DEBUG -- : lib/oxidized/model/model.rb Added secret to the commands list                                                                                                                              
D, [2016-09-05T12:33:59.405409 #51453] DEBUG -- : lib/oxidized/model/model.rb Added show version to the commands list                                                                                                                        
D, [2016-09-05T12:33:59.405463 #51453] DEBUG -- : lib/oxidized/model/model.rb Added show inventory to the commands list                                                                                                                      
D, [2016-09-05T12:33:59.405544 #51453] DEBUG -- : lib/oxidized/model/model.rb Added show running-config to the commands list                                                                                                                 
I, [2016-09-05T12:33:59.406109 #51453]  INFO -- : lib/oxidized/nodes.rb: Loaded 1 nodes
F, [2016-09-05T12:33:59.478507 #51453] FATAL -- : Oxidized crashed, crashfile written in /root/.config/oxidized/crash
oxidized-web not found: sudo gem install oxidized-web - \
          or disable web support by setting "rest: false" in your configuration
/root/.gem/ruby/gems/oxidized-0.16.3/lib/oxidized/core.rb:22:in `rescue in initialize': oxidized-web not found: sudo gem install oxidized-web - \ (Oxidized::OxidizedError)
          or disable web support by setting "rest: false" in your configuration
        from /root/.gem/ruby/gems/oxidized-0.16.3/lib/oxidized/core.rb:19:in `initialize'
        from /root/.gem/ruby/gems/oxidized-0.16.3/lib/oxidized/core.rb:4:in `new'
        from /root/.gem/ruby/gems/oxidized-0.16.3/lib/oxidized/core.rb:4:in `new'
        from /root/.gem/ruby/gems/oxidized-0.16.3/lib/oxidized/cli.rb:12:in `run'
        from /usr/local/bin/oxidized:9:in `<main>'
ytti commented 8 years ago

What does this say find /root/.gem/ruby/gems/|grep oxidized-web

JfryR commented 8 years ago

nothing appear

if i did find /root/.gem/ruby/gems/|grep oxidized, I see both oxidized-script and oxidized

I will try a local installation by downloading gem files on rubygems

ytti commented 8 years ago

So you seem to have installed 'oxidized' and 'oxidized-web' completely differently. You seem to be running oxidized in non-system directory in /root/.gem/ where as I assume 'oxidized-web' is in /var/lib/gems/X/gems.

Perhaps you've used different 'gem' binary for each.

JfryR commented 8 years ago

I followed instructions and did:

gem install oxidized gem install oxidized-script oxidized-web

When I do "find" on root path it found that: 

# find / -name "*oxidized-web*"
/usr/local/share/gems/gems/oxidized-web-0.7.1
ytti commented 8 years ago

I'm sure you also have oxidized then in /usr/local/share/gems. But it is also in /root/.gem from which you seem to be running it from. And you've not installed oxidized-web there.

Perhaps earlier installation with another gem binary. which oxidized likely is pointing to a version using /root/.gems.

JfryR commented 8 years ago

There is just oxidized-web in it :/ 

# ls /usr/local/share/gems/gems/
oxidized-web-0.7.1

To launch as a service I copied oxidized that were in /root/.gem in /usr/local/bin

which oxidized
/usr/local/bin/oxidized
ytti commented 8 years ago

Do you happen to have *.gem file for either of the package in directory where you installed it? This would cause local installation of ~/root/.gem as well.

Anyhow it's not installed correctly, it will take considerable effort to use SSH to your shell box over github comments. If you want, I'm happy to share ssh screen with you and review your installation, you can use tmate to share ssh http://tmate.ip.fi/

JfryR commented 8 years ago

Thank you a lot, but I can't let you acceed to this server ..

I uninstall everything, download gems about (oxidized oxidized-script oxidized-web) and install them with --local option.

Now all of them are in /usr/local/share/gems/gems

But nothing new, when I want to launch it with REST enable 

oxidized
I, [2016-09-05T15:23:41.042577 #33078]  INFO -- : Oxidized starting, running as pid 33078
I, [2016-09-05T15:23:41.043367 #33078]  INFO -- : lib/oxidized/nodes.rb: Loading nodes
I, [2016-09-05T15:23:41.140377 #33078]  INFO -- : lib/oxidized/nodes.rb: Loaded 1 nodes
F, [2016-09-05T15:23:41.210665 #33078] FATAL -- : Oxidized crashed, crashfile written in /root/.config/oxidized/crash
oxidized-web not found: sudo gem install oxidized-web - \
          or disable web support by setting "rest: false" in your configuration
/usr/local/share/gems/gems/oxidized-0.16.3/lib/oxidized/core.rb:22:in `rescue in initialize': oxidized-web not found: sudo gem install oxidized-web - \ (Oxidized::OxidizedError)
          or disable web support by setting "rest: false" in your configuration
        from /usr/local/share/gems/gems/oxidized-0.16.3/lib/oxidized/core.rb:19:in `initialize'
        from /usr/local/share/gems/gems/oxidized-0.16.3/lib/oxidized/core.rb:4:in `new'
        from /usr/local/share/gems/gems/oxidized-0.16.3/lib/oxidized/core.rb:4:in `new'
        from /usr/local/share/gems/gems/oxidized-0.16.3/lib/oxidized/cli.rb:12:in `run'
        from /usr/local/share/gems/gems/oxidized-0.16.3/bin/oxidized:9:in `<top (required)>'
        from /usr/local/bin/oxidized:23:in `load'
        from /usr/local/bin/oxidized:23:in `<main>'
ytti commented 8 years ago

Dunno what to tell you.

/usr/local/share/gems/gems looks weird to me, but as long as both oxidized and oxidized-web are in same directory structure, and have correct file permissions. It should work.

Unfortunately I feel we're at point of diminishing returns in our efforts to troubleshoot. I'm sure you are able to solve this by yourself faster than us working together through github issue.

If you can recreate the problem in a server to which you can share your session with me, that would be great. If it's security/contract issue, I can issue you 1USD invoice and sign a contracting gig.

JfryR commented 8 years ago

For sure, I understand. I'm very pleased to see how fast you can reply, thanks for that.

I will investigate, it seems it failed to load puma_http11 (I see it with strace) I saw that there were several same issue on google with Kibana

ytti commented 8 years ago

oxidized-web installs puma and puma will compile puma_http11 during installation

[root@ytti.fi /home/ytti]# find /var/lib/gems|grep -i puma
[1 root@ytti.fi /home/ytti]# find /var/lib/gems|grep -i puma_http11
[1 root@ytti.fi /home/ytti]# gem install puma
Fetching: puma-3.6.0.gem (100%)
Building native extensions.  This could take a while...
Successfully installed puma-3.6.0
Parsing documentation for puma-3.6.0
Installing ri documentation for puma-3.6.0
Done installing documentation for puma after 9 seconds
1 gem installed
[root@ytti.fi /home/ytti]# find /var/lib/gems|grep -i puma_http11
/var/lib/gems/2.3.0/doc/puma-3.6.0/ri/ext/puma_http11
/var/lib/gems/2.3.0/doc/puma-3.6.0/ri/ext/puma_http11/page-Makefile.ri
/var/lib/gems/2.3.0/doc/puma-3.6.0/ri/ext/puma_http11/page-PumaHttp11Service_java.ri
/var/lib/gems/2.3.0/doc/puma-3.6.0/ri/ext/puma_http11/page-http11_parser_java_rl.ri
/var/lib/gems/2.3.0/doc/puma-3.6.0/ri/ext/puma_http11/page-http11_parser_rl.ri
/var/lib/gems/2.3.0/doc/puma-3.6.0/ri/ext/puma_http11/page-http11_parser_common_rl.ri
/var/lib/gems/2.3.0/doc/puma-3.6.0/ri/ext/puma_http11/org
/var/lib/gems/2.3.0/doc/puma-3.6.0/ri/ext/puma_http11/org/jruby
/var/lib/gems/2.3.0/doc/puma-3.6.0/ri/ext/puma_http11/org/jruby/puma
/var/lib/gems/2.3.0/doc/puma-3.6.0/ri/ext/puma_http11/org/jruby/puma/page-Http11_java.ri
/var/lib/gems/2.3.0/doc/puma-3.6.0/ri/ext/puma_http11/org/jruby/puma/page-Http11Parser_java.ri
/var/lib/gems/2.3.0/doc/puma-3.6.0/ri/ext/puma_http11/org/jruby/puma/page-MiniSSL_java.ri
/var/lib/gems/2.3.0/extensions/x86_64-linux/2.3.0/puma-3.6.0/puma/puma_http11.so
/var/lib/gems/2.3.0/gems/puma-3.6.0/ext/puma_http11
/var/lib/gems/2.3.0/gems/puma-3.6.0/ext/puma_http11/PumaHttp11Service.java
/var/lib/gems/2.3.0/gems/puma-3.6.0/ext/puma_http11/ext_help.h
/var/lib/gems/2.3.0/gems/puma-3.6.0/ext/puma_http11/extconf.rb
/var/lib/gems/2.3.0/gems/puma-3.6.0/ext/puma_http11/http11_parser.c
/var/lib/gems/2.3.0/gems/puma-3.6.0/ext/puma_http11/http11_parser.h
/var/lib/gems/2.3.0/gems/puma-3.6.0/ext/puma_http11/http11_parser.java.rl
/var/lib/gems/2.3.0/gems/puma-3.6.0/ext/puma_http11/http11_parser.rl
/var/lib/gems/2.3.0/gems/puma-3.6.0/ext/puma_http11/http11_parser_common.rl
/var/lib/gems/2.3.0/gems/puma-3.6.0/ext/puma_http11/io_buffer.c
/var/lib/gems/2.3.0/gems/puma-3.6.0/ext/puma_http11/mini_ssl.c
/var/lib/gems/2.3.0/gems/puma-3.6.0/ext/puma_http11/org
/var/lib/gems/2.3.0/gems/puma-3.6.0/ext/puma_http11/org/jruby
/var/lib/gems/2.3.0/gems/puma-3.6.0/ext/puma_http11/org/jruby/puma
/var/lib/gems/2.3.0/gems/puma-3.6.0/ext/puma_http11/org/jruby/puma/Http11.java
/var/lib/gems/2.3.0/gems/puma-3.6.0/ext/puma_http11/org/jruby/puma/Http11Parser.java
/var/lib/gems/2.3.0/gems/puma-3.6.0/ext/puma_http11/org/jruby/puma/MiniSSL.java
/var/lib/gems/2.3.0/gems/puma-3.6.0/ext/puma_http11/puma_http11.c
/var/lib/gems/2.3.0/gems/puma-3.6.0/ext/puma_http11/Makefile
/var/lib/gems/2.3.0/gems/puma-3.6.0/ext/puma_http11/http11_parser.o
/var/lib/gems/2.3.0/gems/puma-3.6.0/ext/puma_http11/io_buffer.o
/var/lib/gems/2.3.0/gems/puma-3.6.0/ext/puma_http11/mini_ssl.o
/var/lib/gems/2.3.0/gems/puma-3.6.0/ext/puma_http11/puma_http11.o
/var/lib/gems/2.3.0/gems/puma-3.6.0/ext/puma_http11/puma_http11.so
/var/lib/gems/2.3.0/gems/puma-3.6.0/ext/puma_http11/.RUBYARCHDIR.-.puma.time
/var/lib/gems/2.3.0/gems/puma-3.6.0/lib/puma/puma_http11.so
[root@ytti.fi /home/ytti]#
JfryR commented 8 years ago

After uninstall puma and reinstall oxidized-web, all seems ok. So for now it works.

Thanks for your help