search

rapid7 / metasploit-framework

Metasploit Framework
https://www.metasploit.com/
Other
33.1k stars 13.76k forks source link

Metasploit Community rpc server not starting on Kali 1.1.0a #5055

Closed after100 closed 8 years ago

after100 commented 9 years ago

Typing go_pro does not appear to actually start the appropriate services on Kali Linux.

OS: Kali 1.1.0a x64 Metasploit: Stock & Updated (4.11.1-2015031001 & 4.11.1-2015032401)

Error:

msf > go_pro
[*] Starting the Metasploit services. This can take a little time.
[*] Starting PostgreSQL 9.1 database server: main.
[*] Starting Metasploit rpc server: prosvc.
[*] Metasploit web server already started.
[*] Starting Metasploit worker: worker
[-] Metasploit services aren't running. Type 'service metasploit start' and try again.
msf >

...not starting.

4.11.1-2015032401

root@kali:~# service postgresql start && service metasploit start && msfconsole
[ ok ] Starting PostgreSQL 9.1 database server: main.
[ ok ] Starting Metasploit rpc server: prosvc.
[ ok ] Starting Metasploit web server: thin.
[ ok ] Starting Metasploit worker: worker.
[*] Starting the Metasploit Framework console...-

[...]

msf > version
Framework: 4.11.1-2015032401
Console  : 4.11.1-2015032401.15168
msf > date
[*] exec: date

Wed Apr  1 18:49:16 UTC 2015
msf > go_pro
[*] Starting the Metasploit services. This can take a little time.
[*] Starting PostgreSQL 9.1 database server: main.
[*] Starting Metasploit rpc server: prosvc.
[*] Metasploit web server already started.
[*] Starting Metasploit worker: worker
[-] Metasploit services aren't running. Type 'service metasploit start' and try again.
msf > ps aux | grep -i meta
[*] exec: ps aux | grep -i meta

daemon     6339  0.0  0.0   4192  1412 ?        S    18:40   0:00 /bin/sh /opt/metasploit/apps/pro/ui/scripts/worker_ctl.sh startwait
root       6365 21.7 22.6 645776 462388 pts/0   Sl+  18:40   2:05 /opt/metasploit/ruby/bin/.ruby.bin /opt/metasploit/apps/pro/msf3/msfconsole
daemon     6377  0.1  0.6 107952 12956 ?        Sl   18:40   0:00 /opt/metasploit/ruby/bin/.ruby.bin /opt/metasploit/apps/pro/ui/scripts/worker_ctl.rb startwait
root       8230  0.0  0.0   4192   460 pts/0    S+   18:50   0:00 sh -c ps aux | grep -i meta
root       8232  0.0  0.0   7780  1368 pts/0    S+   18:50   0:00 grep -i meta
msf > ps aux | grep -i msf
[*] exec: ps aux | grep -i msf

root       6365 21.4 22.6 645776 462388 pts/0   Sl+  18:40   2:05 /opt/metasploit/ruby/bin/.ruby.bin /opt/metasploit/apps/pro/msf3/msfconsole
postgres   6426  0.0  0.5 103008 11956 ?        Ss   18:40   0:00 postgres: msf3 msf3 ::1(34405) idle                                                                            
postgres   6430  0.0  0.7 103740 14728 ?        Ss   18:40   0:00 postgres: msf3 msf3 ::1(34409) idle                                                                            
postgres   6459  3.8  1.1 104976 23004 ?        Ss   18:40   0:21 postgres: msf3 msf3 ::1(34426) idle                                                                            
postgres   6479  0.0  0.5 103664 12064 ?        Ss   18:40   0:00 postgres: msf3 msf3 ::1(34429) idle                                                                            
root       8260  0.0  0.0   4192   540 pts/0    S+   18:50   0:00 sh -c ps aux | grep -i msf
root       8262  0.0  0.0   7780  1376 pts/0    S+   18:50   0:00 grep -i msf
msf > 
msf > go_pro
[*] Starting the Metasploit services. This can take a little time.
[*] Starting PostgreSQL 9.1 database server: main.
[*] Starting Metasploit rpc server: prosvc.
[*] Starting Metasploit web server: thin.
[*] Starting Metasploit worker: worker
[-] Metasploit services aren't running. Type 'service metasploit start' and try again.
msf > date
[*] exec: date

Wed Apr  1 18:57:46 UTC 2015
msf > 
msf > go_pro
[*] Starting the Metasploit services. This can take a little time.
[*] Starting PostgreSQL 9.1 database server: main.
[*] Starting Metasploit rpc server: prosvc.
[*] Metasploit web server already started.
[*] Starting Metasploit worker: worker.
[-] Metasploit services aren't running. Type 'service metasploit start' and try again.
msf > date
[*] exec: date

Wed Apr  1 19:02:26 UTC 2015
msf >
todb-r7 commented 9 years ago

I updated the title to be a little more specific, since this seems to be the ultimate problem here:

root@lys:~# service metasploit status
[FAIL] Metasploit rpc server is not running ... failed!
[ ok ] Metasploit web server is running.
[FAIL] Metasploit worker is not running ... failed!

Verifiable by your descriptions, @after100 . Thanks for those.

todb-r7 commented 9 years ago

/cc @cdoughty-r7 looks like a packaging issue?

todb-r7 commented 9 years ago

Reproduced here, on a 8GB, 4proc Kali VM:

root@lys:~# service metasploit status
[FAIL] Metasploit rpc server is not running ... failed!
[FAIL] Metasploit web server is not running ... failed!
[FAIL] Metasploit worker is not running ... failed!
todb-r7 commented 9 years ago

Say @after100, did you happen to dist-upgrade on Kali?

I'm wondering if there's some fail there. I'm going to try a fresh install from latest ISO to see if it still happens.

todb-r7 commented 9 years ago

FWIW, I'm on Kali 1.0.9, dist-upgrading to 1.1.0. Since the regular "install latest Kali and latest packages" tests haven't come across this in house, I'm betting there's a dependency that's getting bent out of shape on the dist-upgrade.

todb-r7 commented 9 years ago

Okay, freshly dist-upgraded, no other touching, and go_pro works for me on the latest Metasploit packages.

root@lys:~# date && service metasploit status
Wed Apr  1 17:36:26 CDT 2015
[ ok ] Metasploit rpc server is running.
[ ok ] Metasploit web server is running.
[ ok ] Metasploit worker is running.
root@lys:~# lsb_release -a
No LSB modules are available.
Distributor ID: Kali
Description:    Kali GNU/Linux 1.1.0
Release:    1.1.0
Codename:   moto

My last tests, I definitely had some funny local configs while experimenting around with http://r-7.co/MSF-DEV.

@after100, this seems like a local platform problem. Closing as can't reproduce. Do comment here if there's more to this story, though.

after100 commented 9 years ago

Happens to me on both premade kali vm & iso (both x64)

If I upgrade from 1.0.9a to 1.1.0a not fine Fresh install of 1.1.0a - not fine. Fresh install of 1.0.9a - its fine.

after100 commented 9 years ago

Doesn't work:

root@kali:~# service postgresql start && service metasploit start
[ ok ] Starting PostgreSQL 9.1 database server: main.
Configuring Metasploit...
Creating metasploit database user 'msf3'...
Creating metasploit database 'msf3'...
insserv: warning: current start runlevel(s) (empty) of script `metasploit' overrides LSB defaults (2 3 4 5).
insserv: warning: current stop runlevel(s) (0 1 2 3 4 5 6) of script `metasploit' overrides LSB defaults (0 1 6).
[ ok ] Starting Metasploit rpc server: prosvc.
[ ok ] Starting Metasploit web server: thin.
[ ok ] Starting Metasploit worker: worker.
root@kali:~# service metasploit status
[FAIL] Metasploit rpc server is not running ... failed!
[ ok ] Metasploit web server is running.
[FAIL] Metasploit worker is not running ... failed!
root@kali:~# msfconsole -v
Framework Version: 4.11.1-2015031001
root@kali:~# uname -a
Linux kali 3.18.0-kali3-amd64 #1 SMP Debian 3.18.6-1~kali2 (2015-03-02) x86_64 GNU/Linux
root@kali:~# lsb_release -a
No LSB modules are available.
Distributor ID: Kali
Description:    Kali GNU/Linux 1.1.0
Release:    1.1.0
Codename:   moto
root@kali:~# cat /etc/apt/sources.list
# 

# deb cdrom:[Debian GNU/Linux 7.0 _Kali_ - Official Snapshot amd64 LIVE/INSTALL Binary 20150312-17:50]/ kali contrib main non-free

#deb cdrom:[Debian GNU/Linux 7.0 _Kali_ - Official Snapshot amd64 LIVE/INSTALL Binary 20150312-17:50]/ kali contrib main non-free

deb http://http.kali.org/kali kali main non-free contrib
deb-src http://http.kali.org/kali kali main non-free contrib

deb http://security.kali.org/ kali/updates main contrib non-free
deb-src http://security.kali.org/ kali/updates main contrib non-free
root@kali:~#
after100 commented 9 years ago

Doesn't work

after100 commented 9 years ago

It works

root@kali:~# service postgresql start && service metasploit start
[ ok ] Starting PostgreSQL 9.1 database server: main.
Configuring Metasploit...
Creating metasploit database user 'msf3'...
Creating metasploit database 'msf3'...
insserv: warning: current start runlevel(s) (empty) of script `metasploit' overrides LSB defaults (2 3 4 5).
insserv: warning: current stop runlevel(s) (0 1 2 3 4 5 6) of script `metasploit' overrides LSB defaults (0 1 6).
[ ok ] Starting Metasploit rpc server: prosvc.
[ ok ] Starting Metasploit web server: thin.
[ ok ] Starting Metasploit worker: worker.
root@kali:~# service metasploit status
[ ok ] Metasploit rpc server is running.
[ ok ] Metasploit web server is running.
[ ok ] Metasploit worker is running.
root@kali:~# msfconsole -v
Framework Version: 4.10.0-2014100101
root@kali:~# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux Kali Linux 1.0.9
Release:    Kali Linux 1.0.9
Codename:   n/a
root@kali:~# uname -a
Linux kali 3.14-kali1-amd64 #1 SMP Debian 3.14.5-1kali1 (2014-06-07) x86_64 GNU/Linux
root@kali:~# cat /etc/apt/sources.list
# 

# deb cdrom:[Debian GNU/Linux 7.0 _Kali_ - Official Snapshot amd64 LIVE/INSTALL Binary 20141002-11:29]/ kali contrib main non-free

#deb cdrom:[Debian GNU/Linux 7.0 _Kali_ - Official Snapshot amd64 LIVE/INSTALL Binary 20141002-11:29]/ kali contrib main non-free

deb http://http.kali.org/kali kali main non-free contrib
deb-src http://http.kali.org/kali kali main non-free contrib

## Security updates
deb http://security.kali.org/kali-security kali/updates main contrib non-free
root@kali:~#
after100 commented 9 years ago

Doesn't work.

todb-r7 commented 9 years ago

Comment from #5060 from @after100, the original reporter:

bump issue #5055 reopen at @todb-r7's request.

ive added more detail - able todo it every time on a clean 1.0.9a & 1.1.0a. bug is still there!!!

i shouldnt be so scared at running apt-get dist-upgrade and seeing an update to the metasploit package this isnt the first 'bug' ive had. or the kali community there was the whole issue of wanting ruby 2, but kali only had ruby 1.9.x... before then there was the database upgrade from 4.10 to 4.11 and messing up postgresql

i would take a stable bi-weekly/monthly update, rather than what feels to be a 'bleeding' edge weekly update. if i did what that i would git clone rather than use the repository package

i get its a open source & free project.. but there is a commercial side, msf pro. kali is meant to be the os that you (rapid 7) guys support with it. seems far from it. feels like theres little (or non) testing on the packages

on to that the fact i reported the bug. then someone reported they are able to replicate it. then they couldnt & closes it. feels odd.

happy to give any more information, if i knew where to look. i just dont see any documentation on troubleshooting where log files are stored/what to check where stuff isnt working. true of any open source project, shouldnt be with commercial.

todb-r7 commented 9 years ago

So, it looks like the tldr of all this is this comment . @after100 is seeing issues on 1.1.0a, on both fresh install and on dist-upgrade.

From what I can see, your dist-upgrade does look fine, but that shouldn't matter anyway.

Also, FWIW, I seemed to be running into similar problems, but a VM reset and a redo of the of the dist-upgrade cleared things up, so it doesn't appear to be a packaging or a code issue. That's why I closed the bug initially, as this is not a bug tracker for general support, but specifically for bugs and feature requests.

That said, I bet we could handle failures more gracefully. At least, provide some guidance of where relevant logs live. Also, the usual community channels are offline at the moment, though the metasploit Freenode channel is still a fine place to ask for assistance.

All that said:

How much memory to you have devoted to your Kali machine? How many processors? And what happens when you type service metasploit status ? If you're low on resources (often the case right after a dist-upgrade), that initial startup can take a looong time.

kgray-r7 commented 9 years ago

A few notes on Kali and service commands: If you run service metasploit start and run service metasploit status before those services are up and running, their status will be failed. Metasploit rpc server and Metasploit worker both can take a while to start (worker taking the longest) and until both of these are started, the go_pro command will most likely time out and return the error you're seeing.

Feel free to use the service metasploit status check until the status is ok for all three processes, then use the go_pro command from msfconsole.

Finally, there is an issue with using service metasploit start if the services have not shut down which causes the rpc server to fail. Please use service metasploit stop before running service metasploit start again (or service metasploit restart will work as well.) I also believe the go_pro command will run the service metasploit start command if the services are not fully running, which will cause this issue as well. We're looking into this now.

after100 commented 9 years ago

"as this is not a bug tracker for general support, but specifically for bugs and feature requests." not a bug tracker... but bugs..........???

"That said, I bet we could handle failures more gracefully." not sure if this is about this thread, or the metasploit project itself

"At least, provide some guidance of where relevant logs live. Also, the usual community channels are offline at the moment, though the metasploit Freenode channel is still a fine place to ask for assistance." so where do the logs live? never did answer the question irc sucks for me - its real time and timezones (im not usa so MOST people are sleeping) get in the way. only works when people are at keyboard. irc is also a 'one time thing' - you have to type it out each time. much better to write it up once and then just point people to it. up side is google will also index it and the people that are able to search and find it for themselves. when your community is back, would i find my questions there already - eg where are logs? what todo when things go wrong?

"How much memory to you have devoted to your Kali machine? How many processors?" 6GB. 2 CPU. works fine all up till 1.1.0a hell, even nessus works

"And what happens when you type service metasploit status ?" see all the output that ive pasted. covered it already

"If you're low on resources (often the case right after a dist-upgrade), that initial startup can take a looong time." waited 10 mins. nothing watching on top. nothing is higher than 1.2% CPU. restarted the box (so you can't say dist-upgrade is the issue). try again nothing. waited 10 mins. nothing watching on top. nothing.

"A few notes on Kali and service commands: If you run service metasploit start and run service metasploit status before those services are up and running, their status will be failed." yup. covered it already. see the output

"Finally, there is an issue with using service metasploit start if the services have not shut down which causes the rpc server to fail. Please use service metasploit stop before running service metasploit start again (or service metasploit restart will work as well.) I also believe the go_pro command will run the service metasploit start command if the services are not fully running, which will cause this issue as well. We're looking into this now." so that all to me does sound like a package issue.

after100 commented 9 years ago

stage 1

service metasploit status

  Metasploit rpc server is not running ... failed!

stage 2

file: /etc/init.d/metasploit

  INSTDIR=/opt/metasploit
  SETENV="$INSTDIR/scripts/setenv.sh"
  PROSVC_SCRIPT="$INSTDIR/apps/pro/engine/scripts/ctl.sh"
  PROSVC_PIDFILE="$INSTDIR/apps/pro/engine/tmp/prosvc.pid"
  ...
  $PROSVC_SCRIPT start >/dev/null

stage 3

/opt/metasploit/apps/pro/engine/scripts/ctl.sh start

  prosvc is running

file: /opt/metasploit/apps/pro/engine/scripts/ctl.sh

  ruby "${BASE}/ctl.rb" $@

/opt/metasploit/apps/pro/engine/scripts/ctl.rb

  when "start"
      Dir.chdir(File.join(@pro, "apps", "pro", "engine"))
      system("nohup ruby prosvc.rb -E production >prosvc_stdout.log 2>prosvc_stderr.log &")
      puts "prosvc is running"

...ooo look. log files for this service.

stage 4

find / -name 'prosvc_*.log'

  $ find / -name 'prosvc_*.log'
  /opt/metasploit/apps/pro/engine/prosvc_stderr.log
  /opt/metasploit/apps/pro/engine/prosvc_stdout.log
  $

stage 5

$ cat /opt/metasploit/apps/pro/engine/prosvc_stderr.log
/opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/activerecord-3.2.21/lib/active_record/connection_adapters/postgresql_adapter.rb:1163:in `async_exec': PG::UndefinedColumn: ERROR:  column tasks.state does not exist (ActiveRecord::StatementInvalid)
LINE 1: SELECT "tasks".* FROM "tasks"  WHERE "tasks"."state" = 'runn...
                                             ^
: SELECT "tasks".* FROM "tasks"  WHERE "tasks"."state" = 'running'
    from /opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/activerecord-3.2.21/lib/active_record/connection_adapters/postgresql_adapter.rb:1163:in `exec_no_cache'
    from /opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/activerecord-3.2.21/lib/active_record/connection_adapters/postgresql_adapter.rb:660:in `block in exec_query'
    from /opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract_adapter.rb:280:in `block in log'
    from /opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/activesupport-3.2.21/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
    from /opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract_adapter.rb:275:in `log'
    from /opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/activerecord-3.2.21/lib/active_record/connection_adapters/postgresql_adapter.rb:659:in `exec_query'
    from /opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/activerecord-3.2.21/lib/active_record/connection_adapters/postgresql_adapter.rb:1262:in `select'
    from /opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract/database_statements.rb:18:in `select_all'
    from /opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract/query_cache.rb:63:in `select_all'
    from /opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/activerecord-3.2.21/lib/active_record/querying.rb:38:in `block in find_by_sql'
    from /opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/activerecord-3.2.21/lib/active_record/explain.rb:41:in `logging_query_plan'
    from /opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/activerecord-3.2.21/lib/active_record/querying.rb:37:in `find_by_sql'
    from /opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/activerecord-3.2.21/lib/active_record/relation.rb:171:in `exec_queries'
    from /opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/activerecord-3.2.21/lib/active_record/relation.rb:160:in `block in to_a'
    from /opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/activerecord-3.2.21/lib/active_record/explain.rb:41:in `logging_query_plan'
    from /opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/activerecord-3.2.21/lib/active_record/relation.rb:159:in `to_a'
    from /opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/activerecord-3.2.21/lib/active_record/relation/delegation.rb:6:in `each'
    from /opt/metasploit/apps/pro/engine/config/application.rb:154:in `update_defunct_tasks!'
    from /opt/metasploit/apps/pro/engine/config/application.rb:112:in `block in <class:Application>'
    from /opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/railties-3.2.21/lib/rails/initializable.rb:30:in `instance_exec'
    from /opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/railties-3.2.21/lib/rails/initializable.rb:30:in `run'
    from /opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/railties-3.2.21/lib/rails/initializable.rb:55:in `block in run_initializers'
    from /opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/railties-3.2.21/lib/rails/initializable.rb:54:in `each'
    from /opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/railties-3.2.21/lib/rails/initializable.rb:54:in `run_initializers'
    from /opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/railties-3.2.21/lib/rails/application.rb:136:in `initialize!'
    from /opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/railties-3.2.21/lib/rails/railtie/configurable.rb:30:in `method_missing'
    from /opt/metasploit/apps/pro/engine/config/environment.rb:5:in `<top (required)>'
    from /opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
    from /opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
    from /opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/activesupport-3.2.21/lib/active_support/dependencies.rb:251:in `block in require'
    from /opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/activesupport-3.2.21/lib/active_support/dependencies.rb:236:in `load_dependency'
    from /opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/activesupport-3.2.21/lib/active_support/dependencies.rb:251:in `require'
    from /opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/railties-3.2.21/lib/rails/application.rb:103:in `require_environment!'
    from /opt/metasploit/apps/pro/engine/lib/metasploit/pro/engine/command/base.rb:49:in `require_environment!'
    from /opt/metasploit/apps/pro/engine/lib/metasploit/pro/engine/command/base.rb:65:in `start'
    from prosvc.rb:17:in `<main>'
$
$ cat /opt/metasploit/apps/pro/engine/prosvc_stdout.log
$

...looks like a ruby issue & database issue. not a VM settings issue. not me waiting x amount of mintues .

after100 commented 9 years ago

fyi this isn't on a fresh install of kali. gone back to my own version to see if i can figure whats up

stage 1

service metasploit status

  Metasploit worker is not running ... failed!

stage 2

file: /etc/init.d/metasploit

  INSTDIR=/opt/metasploit
  SETENV="$INSTDIR/scripts/setenv.sh"
  WORKER_SCRIPT="$INSTDIR/apps/pro/ui/scripts/worker_ctl.sh"
  ...
  $WORKER_SCRIPT start >/dev/null

stage 3

/opt/metasploit/apps/pro/ui/scripts/worker_ctl.sh start

  Worker starting in background
  Line2: set -x

stage 4

$ service metasploit start                                                                                                                                    
[ ok ] Starting Metasploit rpc server: prosvc.
[ ok ] Starting Metasploit web server: thin.
[....] Starting Metasploit worker: worker+ SOURCE=/opt/metasploit/apps/pro/ui/scripts/worker_ctl.sh
+ [ -h /opt/metasploit/apps/pro/ui/scripts/worker_ctl.sh ]
+ dirname /opt/metasploit/apps/pro/ui/scripts/worker_ctl.sh
+ cd -P /opt/metasploit/apps/pro/ui/scripts
+ pwd
+ BASE=/opt/metasploit/apps/pro/ui/scripts
+ cd /opt/metasploit/apps/pro/ui/scripts
+ id -u
+ [ 0 -eq 0 ]
+ exec su daemon -s /bin/sh -c /opt/metasploit/apps/pro/ui/scripts/worker_ctl.sh start
+ SOURCE=/opt/metasploit/apps/pro/ui/scripts/worker_ctl.sh
+ [ -h /opt/metasploit/apps/pro/ui/scripts/worker_ctl.sh ]
+ dirname /opt/metasploit/apps/pro/ui/scripts/worker_ctl.sh
+ cd -P /opt/metasploit/apps/pro/ui/scripts
+ pwd
+ BASE=/opt/metasploit/apps/pro/ui/scripts
+ cd /opt/metasploit/apps/pro/ui/scripts
+ id -u
+ [ 1 -eq 0 ]
+ . /opt/metasploit/apps/pro/ui/scripts/../../../../scripts/setenv.sh
+ METASPLOIT_ROOT=/opt/metasploit
+ export METASPLOIT_ROOT
+ egrep /opt/metasploit
+ echo /usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ PATH=/opt/metasploit/ruby/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export PATH
+ egrep /opt/metasploit
+ echo /opt/metasploit/ruby/lib:
+ RAILS_ENV=production
+ export RAILS_ENV
+ BUNDLE_GEMFILE=/opt/metasploit/apps/pro/Gemfile
+ export BUNDLE_GEMFILE
+ MSF_DATABASE_CONFIG=/opt/metasploit/apps/pro/ui/config/database.yml
+ export MSF_DATABASE_CONFIG
+ egrep -q /opt/metasploit/apps/pro/vendor/bundle
+ echo /opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0
+ export GEM_PATH
+ [ xstart = xstart ]
+ ruby /opt/metasploit/apps/pro/ui/scripts/worker_ctl.rb status
+ echo Worker starting in background
+ nohup /opt/metasploit/apps/pro/ui/scripts/worker_ctl.sh startwait
. ok 
$

 stage 5

$ /opt/metasploit/apps/pro/ui/scripts/worker_ctl.sh startwait
+ SOURCE=/opt/metasploit/apps/pro/ui/scripts/worker_ctl.sh
+ [ -h /opt/metasploit/apps/pro/ui/scripts/worker_ctl.sh ]
+ dirname /opt/metasploit/apps/pro/ui/scripts/worker_ctl.sh
+ cd -P /opt/metasploit/apps/pro/ui/scripts
+ pwd
+ BASE=/opt/metasploit/apps/pro/ui/scripts
+ cd /opt/metasploit/apps/pro/ui/scripts
+ id -u
+ [ 0 -eq 0 ]
+ exec su daemon -s /bin/sh -c /opt/metasploit/apps/pro/ui/scripts/worker_ctl.sh startwait
+ SOURCE=/opt/metasploit/apps/pro/ui/scripts/worker_ctl.sh
+ [ -h /opt/metasploit/apps/pro/ui/scripts/worker_ctl.sh ]
+ dirname /opt/metasploit/apps/pro/ui/scripts/worker_ctl.sh
+ cd -P /opt/metasploit/apps/pro/ui/scripts
+ pwd
+ BASE=/opt/metasploit/apps/pro/ui/scripts
+ cd /opt/metasploit/apps/pro/ui/scripts
+ id -u
+ [ 1 -eq 0 ]
+ . /opt/metasploit/apps/pro/ui/scripts/../../../../scripts/setenv.sh
+ METASPLOIT_ROOT=/opt/metasploit
+ export METASPLOIT_ROOT
+ echo /usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ egrep /opt/metasploit
+ PATH=/opt/metasploit/ruby/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export PATH
+ echo
+ egrep /opt/metasploit
+ LD_LIBRARY_PATH=/opt/metasploit/ruby/lib:
+ export LD_LIBRARY_PATH
+ RAILS_ENV=production
+ export RAILS_ENV
+ BUNDLE_GEMFILE=/opt/metasploit/apps/pro/Gemfile
+ export BUNDLE_GEMFILE
+ MSF_DATABASE_CONFIG=/opt/metasploit/apps/pro/ui/config/database.yml
+ export MSF_DATABASE_CONFIG
+ echo 
+ egrep -q /opt/metasploit/apps/pro/vendor/bundle
+ GEM_PATH=/opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0
+ export GEM_PATH
+ [ xstartwait = xstart ]
+ ruby /opt/metasploit/apps/pro/ui/scripts/worker_ctl.rb startwait
worker already running
$

suggestion If its ready - say its ready. If its failed - say its failed. if its waiting - say its waiting. not failed.

after100 commented 9 years ago
after100 commented 9 years ago 
$ service metasploit stop
[ ok ] Stopping Metasploit worker: worker.
[ ok ] Stopping Metasploit web server: thin.
[ ok ] Stopping Metasploit rpc server: prosvc.
$ service metasploit start
[ ok ] Starting Metasploit rpc server: prosvc.
[ ok ] Starting Metasploit web server: thin.
[ ok ] Starting Metasploit worker: worker.
$ msfconsole -q
msf > version
Framework: 4.11.1-2015032401
Console  : 4.11.1-2015032401.15168
msf > db_status
[*] postgresql connected to msf3
msf > db_rebuild_cache
[*] Purging and rebuilding the module cache in the background...
msf > sleep 600
msf > db_status
[*] postgresql connected to msf3
msf > go_pro
[*] Starting the Metasploit services. This can take a little time.
[*] Starting PostgreSQL 9.1 database server: main.
[*] Starting Metasploit rpc server: prosvc.
[*] Metasploit web server already started.
[*] Starting Metasploit worker: worker
[-] Metasploit services aren't running. Type 'service metasploit start' and try again.
msf > quit
$ service metasploit stop && service postgresql stop
[ ok ] Stopping Metasploit worker: worker.
[ ok ] Stopping Metasploit web server: thin.
[ ok ] Stopping Metasploit rpc server: prosvc.
[ ok ] Stopping PostgreSQL 9.1 database server: main.
$ service postgresql start
[ ok ] Starting PostgreSQL 9.1 database server: main.
$ sudo -u postgres psql
psql (9.1.15)
Type "help" for help.

postgres=# drop database msf3
postgres=# create database msf3 --owner msf3
postgres-# \q
$ service metasploit start
[ ok ] Starting Metasploit rpc server: prosvc.
[ ok ] Starting Metasploit web server: thin.
[ ok ] Starting Metasploit worker: worker.
$ msfconsole
[*] Starting the Metasploit Framework console...NOTICE:  CREATE TABLE will create implicit sequence "hosts_id_seq" for serial column "hosts.id"
/NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "hosts_pkey" for table "hosts"
[...]

msf > db_status
[*] postgresql connected to postgres
msf > db_rebuild_cache
[*] Purging and rebuilding the module cache in the background...
msf > go_pro
[*] Starting the Metasploit services. This can take a little time.
[*] Starting PostgreSQL 9.1 database server: main.
[*] Starting Metasploit rpc server: prosvc.
[*] Starting Metasploit web server: thin.
[*] Starting Metasploit worker: worker
[-] Metasploit services aren't running. Type 'service metasploit start' and try again.
msf > exit
$ cat /opt/metasploit/apps/pro/engine/prosvc_stderr.log
$ service metasploit status
[FAIL] Metasploit rpc server is not running ... failed!
[FAIL] Metasploit web server is not running ... failed!
[FAIL] Metasploit worker is not running ... failed!
$ service metasploit start
[ ok ] Starting Metasploit rpc server: prosvc.
[ ok ] Starting Metasploit web server: thin.
[....] Starting Metasploit worker: worker
$ cat /opt/metasploit/apps/pro/engine/prosvc_stderr.log
$ service metasploit status
[FAIL] Metasploit rpc server is not running ... failed!
[FAIL] Metasploit web server is not running ... failed!
[FAIL] Metasploit worker is not running ... failed!
$
todb-r7 commented 9 years ago

Thanks for chasing this down, @after100. Again, looks like thr bug here is around handling the failed and in progress start; if you start the services then try go_pro, which tries to start the services again, you end up dooming the RPC server. So, we should have a more useful way to indicate status.

Btw, the IRC channel is peopled by insomniacs from several time zones. Often there's someone around there for more real time troubleshooting. I also hate it when things like that disappear into the mists of history, so hopefully the end result of discussions like that end up here in an issue or PR.

todb-r7 commented 9 years ago

@kgray-r7 does that summation check out? If so we should harrass @shuckins-r7 and @cdoughty-r7 for a packaging fix for Kali to avoid this situation.

kgray-r7 commented 9 years ago

@todb-r7 Yes, your summation is correct. @shuckins-r7 and @cdoughty-r7 have be alerted to this.

todb-r7 commented 9 years ago

@cdoughty-r7 , I'm going to write a real JIRA ticket for this (internally) so we can get this resolved. I'll also edit down these logs quite a bit so the history on this issue is readable without hurting your PgDn finger.

todb-r7 commented 9 years ago

Okay, @cdoughty-r7 , I have GES-2797 moved over now.

after100 commented 9 years ago

u get yelled at for not having enough info.... ...and you get yelled at for giving too much. no plzing people.

OJ commented 9 years ago

/me checks the thread.

Nope, no complaints about too much information.

after100 commented 9 years ago

"so the history on this issue is readable without hurting your PgDn finger."

OJ commented 9 years ago

That's not a jab. That's @todb-r7's sense of humour. Trust me, he's grateful for the extensive information you've provided and is just going to cut down to the bits that highlight the issue.

cdoughty-r7 commented 9 years ago

Thanks @todb-r7 , I will be working on a fix for the issue.

cdoughty-r7 commented 9 years ago

A fix has been created and is being tested. Once its made it into a weekly release, I'll update this ticket with the specific kali metasploit build to use.

howdini commented 9 years ago

Great, we are eagerly awaiting the solution to this problem.

after100 commented 9 years ago

looks like we are not the only ones............ https://gist.github.com/noxferatu/81d9b8f6a8560a2a6a98

cdoughty-r7 commented 9 years ago

The fix is now included in the latest weekly release build on Kali : 4.11.1-2015042001-1kali0.

cr0bar commented 9 years ago

I'm still experiencing similar issues even after updating to the latest build:

root@hacker:~# date && service metasploit status
Thu Apr 23 19:06:13 UTC 2015
[ ok ] Metasploit rpc server is running.
[ ok ] Metasploit web server is running.
[ ok ] Metasploit worker is running.

root@hacker:~# date && service metasploit status
Thu Apr 23 19:06:49 UTC 2015
[FAIL] Metasploit rpc server is not running ... failed!
[ ok ] Metasploit web server is running.
[ ok ] Metasploit worker is running.
       =[ metasploit v4.11.1-2015042001 [core:4.11.1.pre.2015042001 api:1.0.0]]
msf > go_pro
[*] Starting the Metasploit services. This can take a little time.
[*] Starting PostgreSQL 9.1 database server: main.
[*] Starting Metasploit rpc server: prosvc.
[*] Metasploit web server already started.
[*] Metasploit worker already started.
........................................................................................................................................................................................................
[!] For some reason, Community / Pro didn't start in a timely fashion.
[!] You might want to restart the Metasploit services by typing
[!] 'service metasploit restart'. Sorry it didn't work out.
root@hacker:~# date && service metasploit status
Thu Apr 23 19:23:57 UTC 2015
[FAIL] Metasploit rpc server is not running ... failed!
[FAIL] Metasploit web server is not running ... failed!
[ ok ] Metasploit worker is running.
root@hacker:~# uname -a
Linux hacker 3.18.9-v7 #1 SMP PREEMPT Sat Mar 7 07:12:24 EST 2015 armv7l GNU/Linux
root@hacker:~# 
root@hacker:~# lsb_release -a
No LSB modules are available.
Distributor ID: Kali
Description:    Kali GNU/Linux 1.1.0
Release:    1.1.0
Codename:   moto
root@hacker:~# date && service metasploit status
Thu Apr 23 19:35:07 UTC 2015
[FAIL] Metasploit rpc server is not running ... failed!
[FAIL] Metasploit web server is not running ... failed!
[FAIL] Metasploit worker is not running ... failed!
after100 commented 9 years ago

yeah. same here. lol not sure if to reply here, or make a new ticket or just give up

bcook-r7 commented 9 years ago

If there are still issues, we can reopen and look again. The fixes will have made it to the released build by now, correct?

after100 commented 9 years ago

yup it is in kali repos currently re-installing a fresh kali 1.1.0a vm to see if that helps + clean log files.

bcook-r7 commented 9 years ago

I'll give a fresh kali install a try as well, thanks. Will reopen while we do some extra checking.

after100 commented 9 years ago

from fresh kali 1.1.0a x64 install to apt-get dist-upgrade go_pro works upgrading from before it doesnt work

after100 commented 9 years ago 
root@kali:~# apt-get install metasploit metasploit-framework
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following held packages will be changed:
  metasploit metasploit-framework
The following packages will be upgraded:
  metasploit metasploit-framework
2 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 227 MB of archives.
After this operation, 301 kB of additional disk space will be used.
Do you want to continue [Y/n]? y
Get:1 http://http.kali.org/kali/ kali/main metasploit-framework amd64 4.11.1-2015042001-1kali0 [56.5 MB]
Get:2 http://http.kali.org/kali/ kali/non-free metasploit amd64 4.11.1-2015042001-1kali0 [170 MB]
Fetched 227 MB in 56s (3,982 kB/s)
Reading changelogs... Done
(Reading database ... 381186 files and directories currently installed.)
Preparing to replace metasploit-framework 4.11.1-2015041601-1kali0 (using .../metasploit-framework_4.11.1-2015042001-1kali0_amd64.deb) ...
Unpacking replacement metasploit-framework ...
Preparing to replace metasploit 4.11.1-2015041601-1kali0 (using .../metasploit_4.11.1-2015042001-1kali0_amd64.deb) ...
[ ok ] Stopping Metasploit worker: worker.
[ ok ] Stopping Metasploit web server: thin.
[ ok ] Stopping Metasploit rpc server: prosvc.
Leaving 'diversion of /usr/bin/msfbinscan to /usr/bin/msfbinscan.framework by metasploit'
Leaving 'diversion of /usr/bin/msfcli to /usr/bin/msfcli.framework by metasploit'
Leaving 'diversion of /usr/bin/msfconsole to /usr/bin/msfconsole.framework by metasploit'
Leaving 'diversion of /usr/bin/msfd to /usr/bin/msfd.framework by metasploit'
Leaving 'diversion of /usr/bin/msfelfscan to /usr/bin/msfelfscan.framework by metasploit'
Leaving 'diversion of /usr/bin/msfencode to /usr/bin/msfencode.framework by metasploit'
Leaving 'diversion of /usr/bin/msfmachscan to /usr/bin/msfmachscan.framework by metasploit'
Leaving 'diversion of /usr/bin/msfpayload to /usr/bin/msfpayload.framework by metasploit'
Leaving 'diversion of /usr/bin/msfpescan to /usr/bin/msfpescan.framework by metasploit'
Leaving 'diversion of /usr/bin/msfrop to /usr/bin/msfrop.framework by metasploit'
Leaving 'diversion of /usr/bin/msfrpc to /usr/bin/msfrpc.framework by metasploit'
Leaving 'diversion of /usr/bin/msfrpcd to /usr/bin/msfrpcd.framework by metasploit'
Leaving 'diversion of /usr/bin/msfupdate to /usr/bin/msfupdate.framework by metasploit'
Leaving 'diversion of /usr/bin/msfvenom to /usr/bin/msfvenom.framework by metasploit'
Unpacking replacement metasploit ...
Setting up metasploit-framework (4.11.1-2015042001-1kali0) ...
Setting up metasploit (4.11.1-2015042001-1kali0) ...
Installing new version of config file /etc/init.d/metasploit ...
skipping metasploit initialization: postgres not running
insserv: warning: current start runlevel(s) (empty) of script `metasploit' overrides LSB defaults (2 3 4 5).
insserv: warning: current stop runlevel(s) (0 1 2 3 4 5 6) of script `metasploit' overrides LSB defaults (0 1 6).
root@kali:~# service postgresql start
[ ok ] Starting PostgreSQL 9.1 database server: main.
root@kali:~# service metasploit start
[ ok ] Starting Metasploit rpc server: prosvc.
[ ok ] Starting Metasploit web server: thin.
[ ok ] Starting Metasploit worker: worker.
root@kali:~# service metasploit status
[ ok ] Metasploit rpc server is running.
[ ok ] Metasploit web server is running.
[ ok ] Metasploit worker is running.
root@kali:~# msfconsole
[*] Starting the Metasploit Framework console...NOTICE:  CREATE TABLE will create implicit sequence "automatic_exploitation_matches_id_seq" for serial column "automatic_exploitation_matches.id"
\NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "automatic_exploitation_matches_pkey" for table "automatic_exploitation_matches"
NOTICE:  CREATE TABLE will create implicit sequence "automatic_exploitation_match_sets_id_seq" for serial column "automatic_exploitation_match_sets.id"
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "automatic_exploitation_match_sets_pkey" for table "automatic_exploitation_match_sets"
\NOTICE:  CREATE TABLE will create implicit sequence "automatic_exploitation_runs_id_seq" for serial column "automatic_exploitation_runs.id"
/NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "automatic_exploitation_runs_pkey" for table "automatic_exploitation_runs"
NOTICE:  CREATE TABLE will create implicit sequence "automatic_exploitation_match_results_id_seq" for serial column "automatic_exploitation_match_results.id"
|NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "automatic_exploitation_match_results_pkey" for table "automatic_exploitation_match_results"
NOTICE:  CREATE TABLE will create implicit sequence "module_runs_id_seq" for serial column "module_runs.id"
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "module_runs_pkey" for table "module_runs"
-

Unable to handle kernel NULL pointer dereference at virtual address 0xd34db33f
EFLAGS: 00010046
eax: 00000001 ebx: f77c8c00 ecx: 00000000 edx: f77f0001
esi: 803bf014 edi: 8023c755 ebp: 80237f84 esp: 80237f60
ds: 0018   es: 0018  ss: 0018
Process Swapper (Pid: 0, process nr: 0, stackpage=80377000)

Stack: 90909090990909090990909090
       90909090990909090990909090
       90909090.90909090.90909090
       90909090.90909090.90909090
       90909090.90909090.09090900
       90909090.90909090.09090900
       ..........................
       cccccccccccccccccccccccccc
       cccccccccccccccccccccccccc
       ccccccccc.................
       cccccccccccccccccccccccccc
       cccccccccccccccccccccccccc
       .................ccccccccc
       cccccccccccccccccccccccccc
       cccccccccccccccccccccccccc
       ..........................
       ffffffffffffffffffffffffff
       ffffffff..................
       ffffffffffffffffffffffffff
       ffffffff..................
       ffffffff..................
       ffffffff..................

Code: 00 00 00 00 M3 T4 SP L0 1T FR 4M 3W OR K! V3 R5 I0 N4 00 00 00 00
Aiee, Killing Interrupt handler
Kernel panic: Attempted to kill the idle task!
In swapper task - not syncing

Save 45% of your time on large engagements with Metasploit Pro
Learn more on http://rapid7.com/metasploit

       =[ metasploit v4.11.1-2015042001 [core:4.11.1.pre.2015042001 api:1.0.0]]
+ -- --=[ 1445 exploits - 821 auxiliary - 230 post        ]
+ -- --=[ 377 payloads - 37 encoders - 8 nops             ]
+ -- --=[ Free Metasploit Pro trial: http://r-7.co/trymsp ]

msf > go_pro
[*] Starting the Metasploit services. This can take a little time.
[*] Starting PostgreSQL 9.1 database server: main.
[*] Starting Metasploit rpc server: prosvc.
[*] Starting Metasploit web server: thin.
[*] Metasploit worker already started.
......................................................................................................................................................................................................  ..
[!] For some reason, Community / Pro didn't start in a timely fashion.
[!] You might want to restart the Metasploit services by typing
[!] 'service metasploit restart'. Sorry it didn't work out.
msf >   exit
root@kali:~# service metasploit status
[FAIL] Metasploit rpc server is not running ... failed!
[FAIL] Metasploit web server is not running ... failed!
[FAIL] Metasploit worker is not running ... failed!
root@kali:~# cat .msf4/logs/*
[04/23/2015 22:19:09] [e(0)] core: Connection not established: ActiveRecord::ConnectionNotEstablished ActiveRecord::ConnectionNotEstablished:
/opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract/connection_specification.rb:167:in `connection_pool'
/usr/share/metasploit-framework/lib/msf/core/db_manager/connection.rb:123:in `connection_established?'
/usr/share/metasploit-framework/lib/msf/core/db_manager/adapter.rb:32:in `initialize_adapter'
/usr/share/metasploit-framework/lib/msf/core/db_manager.rb:162:in `initialize_database_support'
/usr/share/metasploit-framework/lib/msf/core/db_manager.rb:125:in `initialize'
/usr/share/metasploit-framework/lib/msf/core/framework.rb:197:in `new'
/usr/share/metasploit-framework/lib/msf/core/framework.rb:197:in `block in db'
/opt/metasploit/ruby/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
/usr/share/metasploit-framework/lib/msf/core/framework.rb:196:in `db'
/usr/share/metasploit-framework/lib/msf/ui/console/driver.rb:113:in `initialize'
/usr/share/metasploit-framework/lib/metasploit/framework/command/console.rb:52:in `new'
/usr/share/metasploit-framework/lib/metasploit/framework/command/console.rb:52:in `driver'
/usr/share/metasploit-framework/lib/metasploit/framework/command/console.rb:38:in `start'
/usr/share/metasploit-framework/lib/metasploit/framework/command/base.rb:82:in `start'
/opt/metasploit/apps/pro/msf3/msfconsole:48:in `<main>'
[04/23/2015 22:19:09] [e(0)] core: Connection not established: ActiveRecord::ConnectionNotEstablished ActiveRecord::ConnectionNotEstablished:
/opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract/connection_specification.rb:167:in `connection_pool'
/usr/share/metasploit-framework/lib/msf/core/db_manager/connection.rb:123:in `connection_established?'
/usr/share/metasploit-framework/lib/msf/ui/console/driver.rb:165:in `initialize'
/usr/share/metasploit-framework/lib/metasploit/framework/command/console.rb:52:in `new'
/usr/share/metasploit-framework/lib/metasploit/framework/command/console.rb:52:in `driver'
/usr/share/metasploit-framework/lib/metasploit/framework/command/console.rb:38:in `start'
/usr/share/metasploit-framework/lib/metasploit/framework/command/base.rb:82:in `start'
/opt/metasploit/apps/pro/msf3/msfconsole:48:in `<main>'
[04/23/2015 22:19:09] [e(0)] core: Connection not established: ActiveRecord::ConnectionNotEstablished ActiveRecord::ConnectionNotEstablished:
/opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract/connection_specification.rb:167:in `connection_pool'
/usr/share/metasploit-framework/lib/msf/core/db_manager/connection.rb:123:in `connection_established?'
/usr/share/metasploit-framework/lib/msf/core/db_manager/connection.rb:54:in `connect'
/usr/share/metasploit-framework/lib/msf/ui/console/driver.rb:182:in `initialize'
/usr/share/metasploit-framework/lib/metasploit/framework/command/console.rb:52:in `new'
/usr/share/metasploit-framework/lib/metasploit/framework/command/console.rb:52:in `driver'
/usr/share/metasploit-framework/lib/metasploit/framework/command/console.rb:38:in `start'
/usr/share/metasploit-framework/lib/metasploit/framework/command/base.rb:82:in `start'
/opt/metasploit/apps/pro/msf3/msfconsole:48:in `<main>'
[04/23/2015 22:22:14] [e(0)] core: /usr/share/metasploit-framework/modules/auxiliary/server/pxexploit.rb failed to load due to the following error:
Errno::ENOENT
No such file or directory @ rb_sysopen - /usr/share/metasploit-framework/modules/auxiliary/server/pxexploit.rb
Call stack:
/usr/share/metasploit-framework/lib/msf/core/modules/loader/directory.rb:104:in `initialize'
/usr/share/metasploit-framework/lib/msf/core/modules/loader/directory.rb:104:in `open'
/usr/share/metasploit-framework/lib/msf/core/modules/loader/directory.rb:104:in `read_module_content'
(eval):56:in `read_module_content_with_decryption'
/usr/share/metasploit-framework/lib/msf/core/modules/loader/base.rb:136:in `load_module'
/usr/share/metasploit-framework/lib/msf/core/module_manager/cache.rb:89:in `block in load_cached_module'
/usr/share/metasploit-framework/lib/msf/core/module_manager/cache.rb:84:in `each'
/usr/share/metasploit-framework/lib/msf/core/module_manager/cache.rb:84:in `load_cached_module'
/usr/share/metasploit-framework/lib/msf/core/module_set.rb:45:in `create'
/usr/share/metasploit-framework/lib/msf/core/db_manager/module_cache.rb:360:in `block (3 levels) in update_all_module_details'
/usr/share/metasploit-framework/lib/msf/core/db_manager/module_cache.rb:358:in `each'
/usr/share/metasploit-framework/lib/msf/core/db_manager/module_cache.rb:358:in `block (2 levels) in update_all_module_details'
/usr/share/metasploit-framework/lib/msf/core/db_manager/module_cache.rb:355:in `each'
/usr/share/metasploit-framework/lib/msf/core/db_manager/module_cache.rb:355:in `block in update_all_module_details'
/opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract/connection_pool.rb:129:in `with_connection'
/usr/share/metasploit-framework/lib/msf/core/db_manager/module_cache.rb:318:in `update_all_module_details'
/usr/share/metasploit-framework/lib/msf/core/module_manager/cache.rb:113:in `refresh_cache_from_module_files'
/usr/share/metasploit-framework/lib/msf/ui/console/driver.rb:225:in `block in initialize'
/usr/share/metasploit-framework/lib/msf/core/thread_manager.rb:100:in `call'
/usr/share/metasploit-framework/lib/msf/core/thread_manager.rb:100:in `block in spawn'
[04/23/2015 22:22:20] [e(0)] core: /usr/share/metasploit-framework/modules/post/windows/manage/pxexploit.rb failed to load due to the following error:
Errno::ENOENT
No such file or directory @ rb_sysopen - /usr/share/metasploit-framework/modules/post/windows/manage/pxexploit.rb
Call stack:
/usr/share/metasploit-framework/lib/msf/core/modules/loader/directory.rb:104:in `initialize'
/usr/share/metasploit-framework/lib/msf/core/modules/loader/directory.rb:104:in `open'
/usr/share/metasploit-framework/lib/msf/core/modules/loader/directory.rb:104:in `read_module_content'
(eval):56:in `read_module_content_with_decryption'
/usr/share/metasploit-framework/lib/msf/core/modules/loader/base.rb:136:in `load_module'
/usr/share/metasploit-framework/lib/msf/core/module_manager/cache.rb:89:in `block in load_cached_module'
/usr/share/metasploit-framework/lib/msf/core/module_manager/cache.rb:84:in `each'
/usr/share/metasploit-framework/lib/msf/core/module_manager/cache.rb:84:in `load_cached_module'
/usr/share/metasploit-framework/lib/msf/core/module_set.rb:45:in `create'
/usr/share/metasploit-framework/lib/msf/core/db_manager/module_cache.rb:360:in `block (3 levels) in update_all_module_details'
/usr/share/metasploit-framework/lib/msf/core/db_manager/module_cache.rb:358:in `each'
/usr/share/metasploit-framework/lib/msf/core/db_manager/module_cache.rb:358:in `block (2 levels) in update_all_module_details'
/usr/share/metasploit-framework/lib/msf/core/db_manager/module_cache.rb:355:in `each'
/usr/share/metasploit-framework/lib/msf/core/db_manager/module_cache.rb:355:in `block in update_all_module_details'
/opt/metasploit/apps/pro/vendor/bundle/ruby/2.1.0/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract/connection_pool.rb:129:in `with_connection'
/usr/share/metasploit-framework/lib/msf/core/db_manager/module_cache.rb:318:in `update_all_module_details'
/usr/share/metasploit-framework/lib/msf/core/module_manager/cache.rb:113:in `refresh_cache_from_module_files'
/usr/share/metasploit-framework/lib/msf/ui/console/driver.rb:225:in `block in initialize'
/usr/share/metasploit-framework/lib/msf/core/thread_manager.rb:100:in `call'
/usr/share/metasploit-framework/lib/msf/core/thread_manager.rb:100:in `block in spawn'
cat: .msf4/logs/sessions: Is a directory
root@kali:~#
bcook-r7 commented 9 years ago

Thanks. I tried it a few times to better understand what is happening. This is the behavior of a fresh boot with a fresh kali-linux install and recently dist-upgraded:

root@kali:~# service metasploit status
[FAIL] Metasploit rpc server is not running ... failed!
[FAIL] Metasploit web server is not running ... failed!
[FAIL] Metasploit worker is not running ... failed!
root@kali:~# service postgresql start
[ ok ] Starting PostgreSQL 9.1 database server: main.
root@kali:~# service metasploit start
[ ok ] Starting Metasploit rpc server: prosvc.
[ ok ] Starting Metasploit web server: thin.
[ ok ] Starting Metasploit worker: worker.
root@kali:~# service metasploit status
[ ok ] Metasploit rpc server is running.
[ ok ] Metasploit web server is running.
[ ok ] Metasploit worker is running.

So good so far! However, if you immediately run msfconsole and then go_pro, you will get the failing behavior above, because the metasploit services do not appear to have actually completely started yet. Running go_pro subsequently fails with the long timeout shown above.

However, if you wait a couple of minutes after doing the initial 'service metasploit start' and then start msfconsole, things work perfectly - the go_pro command takes no more than a couple of seconds to run, all works fine.

If you start msfconsole for the first time after running 'service metasploit start' and you get a lot of messages about the database schema being created, that seems to be a sign that go_pro is likely not going to work. I think you end up with pretty much the scenario @kgray-r7 described in his document above.

What are some solutions that can prevent this scenario? It would be reasonable if 'service metasploit status' could report 'starting' if it sees the process running but it is not actually ready yet. Before it said 'failed!' while the service was starting, and now it says 'running' instead, neither of which is really correct. It actually gets a little worse now, because if you run go_pro while metasploit is still starting, you end up with a system that subsequently works like this:

root@kali:~# service metasploit start
[ ok ] Starting Metasploit rpc server: prosvc.
[ ok ] Starting Metasploit web server: thin.
[ ok ] Starting Metasploit worker: worker.
root@kali:~# service metasploit status
[ ok ] Metasploit rpc server is running.
[ ok ] Metasploit web server is running.
[ ok ] Metasploit worker is running.
root@kali:~# sleep 120; service metasploit status
[FAIL] Metasploit rpc server is not running ... failed!
[ ok ] Metasploit web server is running.
[ ok ] Metasploit worker is running.

If we could detect the 'starting' state for the services, perhaps 'msfconsole' could even delay starting, or prevent running 'go_pro' until the services all truly have started.

bcook-r7 commented 9 years ago

Discussed with @cdoughty-r7 today, starting with an empty, default postgresql database and starting msfconsole before the metasploit services have created the initial schema may be the key to reproducing the failure mode.

cr0bar commented 9 years ago

I'm not sure that a delay in the service starting is the issue. At my end when I reboot, all services start correctly. Give it a couple of minutes and the RPC server service will fail, 10 minutes later the Web server service fails and about 30 minutes later, the Worker one does too.

Simply idling will replicate this, no need to fire up msfconsole.

bcook-r7 commented 9 years ago

I think we're on the same page @cr0bar. See above where the log shows the RPC server failing after 2 minutes of idling when I got the install into a bad state.

This 'idle failure' state seems to occur if, at any point in the very beginning of setting up metasploit on Kali linux, if you have started msfconsole and run 'go_pro' while the metasploit services were still starting up and your database was in an empty, uninitialized state. Once in this state, things seemingly will never work again. and services will slowly die as I posted above any time they are restarted.

I'm still working on exact steps to recreate the failure state or to repair it without having to reinstall kali linux each time. I have setup a system that worked perfectly fine without any idle failures as well.

bcook-r7 commented 9 years ago

This is how I was able to provoke either working or broken states from a system already in either state.

First, do this to get the system back into an initial state:

Depending on which state you want to test, perform one of the next series of steps. To recreate a broken system, do the next step immediately:

To have a working system, perform the next step after waiting a couple of minutes for the metasploit service to start:

It might be nice to add a 'service metasploit reinit' that reinitializes the database and starts everything from 'just installed' state to make resetting things easier. Of course, there may already be something like that I just don't know about :)

cr0bar commented 9 years ago

Followed your instructions and now it works fine for me (although I think you meant su postgres rather than su postgresql).

Good work!

cr0bar commented 9 years ago

Spoke too soon.. Fired up the web ui which worked for about a minute, now the rpc service has just failed again..

bcook-r7 commented 9 years ago

I have not been able to reproduce a failure yet once in I get the system into working mode, though I did have issues running everything in general until I bumped my VM to 2GB Ram and 16 GB Disk due to memory pressure. The metasploit services themselves use over 1GB at idle, which can also trigger the OOM killer.

kgray-r7 commented 9 years ago

As this has become a multi-issue ticket, I've created a new issue for the Metasploit RPC server failing to start: https://github.com/rapid7/metasploit-framework/issues/5423

ghost commented 9 years ago

I have exactly the same issue that cr0bar, and the same output, even after bcook-r7's instructions.