search

rapid7 / metasploit-framework

Metasploit Framework
https://www.metasploit.com/
Other
33.83k stars 13.92k forks source link

ruby crash at readline autocomplete #4128

Closed agix closed 9 years ago

agix commented 9 years ago

When finishing autocomplete, in msfconsole shell for files, autocomplete seems to be on $metasploit_path/set $option $path so it trigger a 'No such file or directory'

maybe at least add a rescue to prevent msfconsole to close https://github.com/rapid7/metasploit-framework/blob/master/lib/rex/ui/text/input/readline.rb#L86

msf exploit(psexec) > set EXE::Template /home/agix/http-share/pslist./home/agix/.rvm/gems/ruby-1.9.3-p547@metasploit-framework/gems/rb-readline-0.5.1/lib/rbreadline.rb:8438:in `stat': No such file or directory - /home/agix/git/metasploit-framework/set EXE::Template /home/agix/http-share/pslist.exe (Errno::ENOENT)
        from /home/agix/.rvm/gems/ruby-1.9.3-p547@metasploit-framework/gems/rb-readline-0.5.1/lib/rbreadline.rb:8438:in `append_to_match'
        from /home/agix/.rvm/gems/ruby-1.9.3-p547@metasploit-framework/gems/rb-readline-0.5.1/lib/rbreadline.rb:6812:in `rl_complete_internal'
        from /home/agix/.rvm/gems/ruby-1.9.3-p547@metasploit-framework/gems/rb-readline-0.5.1/lib/rbreadline.rb:6851:in `rl_complete'
        from /home/agix/.rvm/gems/ruby-1.9.3-p547@metasploit-framework/gems/rb-readline-0.5.1/lib/rbreadline.rb:4322:in `_rl_dispatch_subseq'
        from /home/agix/.rvm/gems/ruby-1.9.3-p547@metasploit-framework/gems/rb-readline-0.5.1/lib/rbreadline.rb:4311:in `_rl_dispatch'
        from /home/agix/.rvm/gems/ruby-1.9.3-p547@metasploit-framework/gems/rb-readline-0.5.1/lib/rbreadline.rb:4727:in `readline_internal_charloop'
        from /home/agix/.rvm/gems/ruby-1.9.3-p547@metasploit-framework/gems/rb-readline-0.5.1/lib/rbreadline.rb:4801:in `readline_internal'
        from /home/agix/.rvm/gems/ruby-1.9.3-p547@metasploit-framework/gems/rb-readline-0.5.1/lib/rbreadline.rb:4823:in `readline'
        from /home/agix/.rvm/gems/ruby-1.9.3-p547@metasploit-framework/gems/rb-readline-0.5.1/lib/readline.rb:45:in `readline'
        from /home/agix/git/metasploit-framework/lib/rex/ui/text/input/readline.rb:86:in `pgets'
        from /home/agix/git/metasploit-framework/lib/rex/ui/text/shell.rb:184:in `run'
        from /home/agix/git/metasploit-framework/lib/metasploit/framework/command/console.rb:30:in `start'
        from /home/agix/git/metasploit-framework/lib/metasploit/framework/command/base.rb:82:in `start'
        from ./msfconsole:48:in `<main>'
todb-r7 commented 9 years ago

Hey @agix can you give a hint as to what OS you're seeing this on? I know that when we landed #4084 I tested the tab complete on filenames specifically.

Since you're on a GitHub checkout, you might workaround this with using Ruby 2.1. See #4136 on how to do that easily. Doesn't make it not a bug, but you should at least be able to work around.

agix commented 9 years ago

Okay, I use gentoo and I tried Ruby 2.1 as you suggested but the crash still occurs.

msf exploit(psexec) > set EXE::Template /home/agix/http-share/pslist./home/agix/.rvm/gems/ruby-2.1.4@metasploit-framework/gems/rb-readline-0.5.1/lib/rbreadline.rb:8438:in `stat': No such file or directory @ rb_file_s_stat - /home/agix/git/metasploit-framework/set EXE::Template /home/agix/http-share/pslist.exe (Errno::ENOENT)
    from /home/agix/.rvm/gems/ruby-2.1.4@metasploit-framework/gems/rb-readline-0.5.1/lib/rbreadline.rb:8438:in `append_to_match'
    from /home/agix/.rvm/gems/ruby-2.1.4@metasploit-framework/gems/rb-readline-0.5.1/lib/rbreadline.rb:6812:in `rl_complete_internal'
    from /home/agix/.rvm/gems/ruby-2.1.4@metasploit-framework/gems/rb-readline-0.5.1/lib/rbreadline.rb:6851:in `rl_complete'
    from /home/agix/.rvm/gems/ruby-2.1.4@metasploit-framework/gems/rb-readline-0.5.1/lib/rbreadline.rb:4322:in `_rl_dispatch_subseq'
    from /home/agix/.rvm/gems/ruby-2.1.4@metasploit-framework/gems/rb-readline-0.5.1/lib/rbreadline.rb:4311:in `_rl_dispatch'
    from /home/agix/.rvm/gems/ruby-2.1.4@metasploit-framework/gems/rb-readline-0.5.1/lib/rbreadline.rb:4727:in `readline_internal_charloop'
    from /home/agix/.rvm/gems/ruby-2.1.4@metasploit-framework/gems/rb-readline-0.5.1/lib/rbreadline.rb:4801:in `readline_internal'
    from /home/agix/.rvm/gems/ruby-2.1.4@metasploit-framework/gems/rb-readline-0.5.1/lib/rbreadline.rb:4823:in `readline'
    from /home/agix/.rvm/gems/ruby-2.1.4@metasploit-framework/gems/rb-readline-0.5.1/lib/readline.rb:45:in `readline'
    from /home/agix/git/metasploit-framework/lib/rex/ui/text/input/readline.rb:86:in `pgets'
    from /home/agix/git/metasploit-framework/lib/rex/ui/text/shell.rb:184:in `run'
    from /home/agix/git/metasploit-framework/lib/metasploit/framework/command/console.rb:30:in `start'
    from /home/agix/git/metasploit-framework/lib/metasploit/framework/command/base.rb:82:in `start'
    from ./msfconsole:48:in `<main>'
todb-r7 commented 9 years ago

Reproduced on Ubuntu Linuux.

If I tab complete on a partial file name that does exist, I get a stacktrace. set USERPASS_FILE /tmp/h<tab> (/tmp/here.txt exists) produces:

msf auxiliary(http_login) > set USERPASS_FILE /tmp/her/home/todb/.rvm/gems/ruby-2.1.4@metasploit-framework/gems/rb-readline-0.5.1/lib/rbreadline.rb:8438:in `lstat': No such file or directory @ rb_file_s_lstat - /home/todb/github/metasploit-framework/set USERPASS_FILE /tmp/here.txt (Errno::ENOENT)
        from /home/todb/.rvm/gems/ruby-2.1.4@metasploit-framework/gems/rb-readline-0.5.1/lib/rbreadline.rb:8438:in `append_to_match'
        from /home/todb/.rvm/gems/ruby-2.1.4@metasploit-framework/gems/rb-readline-0.5.1/lib/rbreadline.rb:6812:in `rl_complete_internal'
        from /home/todb/.rvm/gems/ruby-2.1.4@metasploit-framework/gems/rb-readline-0.5.1/lib/rbreadline.rb:6851:in `rl_complete'
        from /home/todb/.rvm/gems/ruby-2.1.4@metasploit-framework/gems/rb-readline-0.5.1/lib/rbreadline.rb:4322:in `_rl_dispatch_subseq'
        from /home/todb/.rvm/gems/ruby-2.1.4@metasploit-framework/gems/rb-readline-0.5.1/lib/rbreadline.rb:4311:in `_rl_dispatch'
        from /home/todb/.rvm/gems/ruby-2.1.4@metasploit-framework/gems/rb-readline-0.5.1/lib/rbreadline.rb:4727:in `readline_internal_charloop'
        from /home/todb/.rvm/gems/ruby-2.1.4@metasploit-framework/gems/rb-readline-0.5.1/lib/rbreadline.rb:4801:in `readline_internal'
        from /home/todb/.rvm/gems/ruby-2.1.4@metasploit-framework/gems/rb-readline-0.5.1/lib/rbreadline.rb:4823:in `readline'
        from /home/todb/.rvm/gems/ruby-2.1.4@metasploit-framework/gems/rb-readline-0.5.1/lib/readline.rb:45:in `readline'
        from /home/todb/github/metasploit-framework/lib/rex/ui/text/input/readline.rb:86:in `pgets'
        from /home/todb/github/metasploit-framework/lib/rex/ui/text/shell.rb:184:in `run'
        from /home/todb/github/metasploit-framework/lib/metasploit/framework/command/console.rb:30:in `start'
        from /home/todb/github/metasploit-framework/lib/metasploit/framework/command/base.rb:82:in `start'
        from ./msfconsole:48:in `<main>'
[2.1.4@metasploit-framework] (upstream-master)

Tab completing on a nonexistant file, a real or nonexistant directory, or any other condition seems not to trigger the issue

I do not trigger the issue.

Same bug and stacktrace as with ruby-2.1.4

Same solution as ruby-2.1.4

As @agix indicated, looks like the problem is in the search path: No such file or directory @ rb_file_s_lstat - /home/todb/github/metasploit-framework/set USERPASS_FILE /tmp/here.txt (Errno::ENOENT)

Looking at the verification steps @jlee-r7 provided, the test was only for non-existant files, which used to be the problem.

That all said -- this is a known issue with some system readlines. Running git revert upstream/pr/4084 and rerunning the repro steps above on both 1.9.3 and 2.1.4 triggers exactly the same issue -- msfconsole -L (system reading) fixes, while msfconsole (no -L) has the same effect.

So, this would appear to be a bug in RbReadline itself, and is a reason why we have a -L argument in the first place. That said, a monkey patch or rescue (as suggested by @agix) looks to be in order to at least avoiding crashing out completely.

todb-r7 commented 9 years ago

Well, crap.

This fix: https://github.com/todb-r7/metasploit-framework/compare/rapid7:master...bug-4128-readline-rescue?expand=1

doesn't fix. All it does is suppress the exception, but msfconsole still immediately exits. I imagine it's because I'm raising in a thread. @kernelsmith can you take a peek? I don't care if broken readline fails to autocomplete, but I do care if you're dumped out without warning. :(

@agix I'd suggest running ./msfconsole -L for now. I'm kinda surprised this hasn't come up before; it's not looking like a new condition (I just always use ./msfconsole -L anyway)

kernelsmith commented 9 years ago

BTW, todb, you’re link opens a new PR. I’m not thinking that’s what you meant

On Nov 7, 2014, at 4:16 PM, Tod Beardsley notifications@github.com wrote:

Well, crap.

This fix: https://github.com/todb-r7/metasploit-framework/compare/rapid7:master...bug-4128-readline-rescue?expand=1

doesn't fix. All it does is suppress the exception, but msfconsole still immediately exits. I imagine it's because I'm raising in a thread. @kernelsmith can you take a peek? I don't care if broken readline fails to autocomplete, but I do care if you're dumped out without warning. :(

@agix I'd suggest running ./msfconsole -L for now. I'm kinda surprised this hasn't come up before; it's not looking like a new condition (I just always use ./msfconsole -L anyway)

— Reply to this email directly or view it on GitHub.

kernelsmith commented 9 years ago

it might be time to roll out my revamped tab completion routines, tho it would be possible I ported the bug too

todb-r7 commented 9 years ago

eh, @kernelsmith, it's the only way I know how to compare. PR it if you want but it's incomplete. :)

kernelsmith commented 9 years ago

Ahh, gotchya tod

On Nov 7, 2014, at 4:30 PM, Tod Beardsley notifications@github.com wrote:

eh, @agix, it's the only way I know how to compare. PR it if you want but it's incomplete. :)

— Reply to this email directly or view it on GitHub.

shuckins-r7 commented 9 years ago

@todb-r7 @kernelsmith Regarding the link above and creating a PR: if you omit the ellipsis you will get a compare with a button to make a PR, which is better for this purpose: https://github.com/todb-r7/metasploit-framework/compare/bug-4128-readline-rescue

kernelsmith commented 9 years ago

Sweet!

On Nov 11, 2014, at 11:11, Samuel Huckins notifications@github.com wrote:

@todb-r7 @kernelsmith Regarding the link above and creating a PR: if you omit the ellipsis you will get a compare with a button to make a PR, which is better for this purpose: https://github.com/todb-r7/metasploit-framework/compare/bug-4128-readline-rescue

— Reply to this email directly or view it on GitHub.

hdm commented 9 years ago

This is an upstream issue, I posted a PR for feedback: https://github.com/luislavena/rb-readline/pull/111

jhart-r7 commented 9 years ago

I just tripped over this bug in an unrelated PR which utilizes OptPath. There looks to be 103 modules which are affected by this.

@todb-r7's suggested fix doesn't fix things (according to him -- I haven't tried), and @hmoore-r7 has a patch that supposedly fixes it upstream.

More and more frequently, the answer seems to be "use -L". Why not simply make this the default, or not even make it an option any more?

hdm commented 9 years ago

@jhart-r7 -L implies readline, which is GPL (boo!) and faked out by libeditline on OS X, which causes background threads to stall. We need to just fix the rbreadline issue.

hdm commented 9 years ago

This is patched in upstream rb-readline master branch. For Metasploit to get this patch, rb-readline needs to release a new gem (metasploit-framework.gemspec references it)

wchen-r7 commented 9 years ago

I'm triggering a similar problem by doing msf > db_import /tmp/t[TAB] (/tmp/test.txt as a valid file), and looks like the latest rb-readline gem 0.5.2 does not actually fix my problem. This gem (0.5.2) was released on December 29th though. It looks like the rb-readline gem is only updated almost once a year or even longer.... so if I have to wait for another year (or longer) to see this patched, then DAMN.

Can you guys suggest what else we can do? Is system Readline actually always better?

agix commented 9 years ago

It seems they didn't apply the provided patch https://github.com/ConnorAtherton/rb-readline/commit/19f0604cec06c677d0963670b919f429c9768cd9 in the new gem

agix@neptune ~/git/metasploit-framework $ sed -n '8435,8439p' < /home/agix/.rvm/gems/ruby-2.1.4@metasploit-framework/gems/rb-readline-0.5.2/lib/rbreadline.rb
    if (@rl_filename_completion_desired)
      filename = File.expand_path(text)
      s = (nontrivial_match && !@rl_completion_mark_symlink_dirs) ?
        File.lstat(filename) : File.stat(filename)
      if s.directory?
hdm commented 9 years ago

Probably worth forking the gem until they fix it.

wchen-r7 commented 9 years ago

It seems they didn't apply the provided patch

ah man...

hdm commented 9 years ago

We are waiting on the vendor to cut a new gem with the fix included (or fork it at this point).

todb-r7 commented 9 years ago

The right thing to do (as discussed on other tickets, Flowdock, and e-mail) is to figure out how to rbreadline correctly, and no longer depend on system readline.

That said, today, starting msfconsole with the -L option will (on Linux systems) use the less crashy system readline, and likely solve the tab completion problems reported here and #4705. But that is a stopgap until we fork and gem rbreadline (and test OSX and Windows support, right @hmoore-r7 ? That's all the steps to get out of readline hell?)

todb-r7 commented 9 years ago

@wvu-r7 is also complaining about missing features.

hdm commented 9 years ago

We can fork and gem rbreadline anytime, a long-term option may be to build a readline gem linked against libeditline.

todb-r7 commented 9 years ago

I checked out the fork of rbreadline, and tried to merge from upstream at https://github.com/rapid7/rb-readline/issues/1 , but I'm sadly not able to, due to merge conflicts.

Should we just refork, @hmoore-r7? It looks like you (or someone) needs to untangle the conflict (maybe caused by to a local set of commits on master, rather than a feature branch?)

hdm commented 9 years ago

@todb-r7 I reforked the repo, changed the gem name, added another revision version (.0), and PRd this change as #4816.

adiov commented 8 years ago

Sadly, I can still reproduce it even with everything updated.

use exploit/windows/browser/ms12_037_same_id /usr/share/metasploit-framework/vendor/bundle/ruby/2.1.0/gems/rb-readline-r7-0.5.2.0/lib/rbreadline.rb:6497:in _rl_internal_pager': private methodputs' called for #Rex::Ui::Text::Output::Tee:0x1037f3ec (NoMethodError) from /usr/share/metasploit-framework/vendor/bundle/ruby/2.1.0/gems/rb-readline-r7-0.5.2.0/lib/rbreadline.rb:8384:in block in rl_display_match_list' from /usr/share/metasploit-framework/vendor/bundle/ruby/2.1.0/gems/activesupport-4.0.13/lib/active_support/core_ext/range/each.rb:7:ineach' from /usr/share/metasploit-framework/vendor/bundle/ruby/2.1.0/gems/activesupport-4.0.13/lib/active_support/core_ext/range/each.rb:7:in each_with_time_with_zone' from /usr/share/metasploit-framework/vendor/bundle/ruby/2.1.0/gems/rb-readline-r7-0.5.2.0/lib/rbreadline.rb:8366:inrl_display_match_list' from /usr/share/metasploit-framework/vendor/bundle/ruby/2.1.0/gems/rb-readline-r7-0.5.2.0/lib/rbreadline.rb:6730:in display_matches' from /usr/share/metasploit-framework/vendor/bundle/ruby/2.1.0/gems/rb-readline-r7-0.5.2.0/lib/rbreadline.rb:6824:inrl_complete_internal' from /usr/share/metasploit-framework/vendor/bundle/ruby/2.1.0/gems/rb-readline-r7-0.5.2.0/lib/rbreadline.rb:6852:in rl_complete' from /usr/share/metasploit-framework/vendor/bundle/ruby/2.1.0/gems/rb-readline-r7-0.5.2.0/lib/rbreadline.rb:4329:in_rl_dispatch_subseq' from /usr/share/metasploit-framework/vendor/bundle/ruby/2.1.0/gems/rb-readline-r7-0.5.2.0/lib/rbreadline.rb:4318:in _rl_dispatch' from /usr/share/metasploit-framework/vendor/bundle/ruby/2.1.0/gems/rb-readline-r7-0.5.2.0/lib/rbreadline.rb:4734:inreadline_internal_charloop' from /usr/share/metasploit-framework/vendor/bundle/ruby/2.1.0/gems/rb-readline-r7-0.5.2.0/lib/rbreadline.rb:4808:in readline_internal' from /usr/share/metasploit-framework/vendor/bundle/ruby/2.1.0/gems/rb-readline-r7-0.5.2.0/lib/rbreadline.rb:4830:inreadline' from /usr/share/metasploit-framework/lib/rex/ui/text/input/readline.rb:132:in readline_with_output' from /usr/share/metasploit-framework/lib/rex/ui/text/input/readline.rb:86:inpgets' from /usr/share/metasploit-framework/lib/rex/ui/text/shell.rb:187:in run' from /usr/share/metasploit-framework/lib/metasploit/framework/command/console.rb:48:instart' from /usr/share/metasploit-framework/lib/metasploit/framework/command/base.rb:82:in start' from /usr/bin/msfconsole:48:in

' root@ak:~# ruby -v ruby 2.1.5p273 (2014-11-13) [i386-linux-gnu] root@ak:~# msfconsole -v Framework Version: 4.11.4-2015090201

hdm commented 8 years ago

This looks like a different issue entirely

hdm commented 8 years ago

New issue filed as #6105.

a14m commented 7 years ago

it still occurs on macOS 10.12 ruby 2.4.1p111 (2017-03-22 revision 58053) [x86_64-darwin16] brew info readline shows readline: stable 7.0.3 (bottled) running ./msfconsole

set pass_file /opt/metasploit-framework[tab]

  metasploit-framework/.bundle/gems/ruby/2.4.0/gems/rb-readline-0.5.4/lib/rbreadline.rb:1401:in `sort': comparison of NilClass with String failed (ArgumentError)
    from metasploit-framework/.bundle/gems/ruby/2.4.0/gems/rb-readline-0.5.4/lib/rbreadline.rb:1401:in `compute_lcd_of_matches'
    from metasploit-framework/.bundle/gems/ruby/2.4.0/gems/rb-readline-0.5.4/lib/rbreadline.rb:1305:in `rl_completion_matches'
    from metasploit-framework/.bundle/gems/ruby/2.4.0/gems/rb-readline-0.5.4/lib/readline.rb:466:in `call'
    from metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:246:in `tab_complete_filenames'
    from metasploit-framework/lib/msf/ui/console/command_dispatcher/core.rb:2246:in `option_values_dispatch'
    from metasploit-framework/lib/msf/ui/console/command_dispatcher/core.rb:2159:in `tab_complete_option'
    from metasploit-framework/lib/msf/ui/console/command_dispatcher/core.rb:1646:in `cmd_set_tabs'
    from metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:360:in `tab_complete_helper'
    from metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:320:in `block in tab_complete_stub'
    from metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:309:in `each'
    from metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:309:in `tab_complete_stub'
    from metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:294:in `tab_complete'
    from metasploit-framework/lib/rex/ui/text/shell.rb:59:in `block in init_tab_complete'
    from metasploit-framework/.bundle/gems/ruby/2.4.0/gems/rb-readline-0.5.4/lib/readline.rb:136:in `readline_attempted_completion_function'
    from metasploit-framework/.bundle/gems/ruby/2.4.0/gems/rb-readline-0.5.4/lib/rbreadline.rb:6329:in `gen_completion_matches'
    from metasploit-framework/.bundle/gems/ruby/2.4.0/gems/rb-readline-0.5.4/lib/rbreadline.rb:6813:in `rl_complete_internal'
    from metasploit-framework/.bundle/gems/ruby/2.4.0/gems/rb-readline-0.5.4/lib/rbreadline.rb:6899:in `rl_complete'
    from metasploit-framework/.bundle/gems/ruby/2.4.0/gems/rb-readline-0.5.4/lib/rbreadline.rb:4374:in `_rl_dispatch_subseq'
    from metasploit-framework/.bundle/gems/ruby/2.4.0/gems/rb-readline-0.5.4/lib/rbreadline.rb:4363:in `_rl_dispatch'
    from metasploit-framework/.bundle/gems/ruby/2.4.0/gems/rb-readline-0.5.4/lib/rbreadline.rb:4779:in `readline_internal_charloop'
    from metasploit-framework/.bundle/gems/ruby/2.4.0/gems/rb-readline-0.5.4/lib/rbreadline.rb:4853:in `readline_internal'
    from metasploit-framework/.bundle/gems/ruby/2.4.0/gems/rb-readline-0.5.4/lib/rbreadline.rb:4875:in `readline'
    from metasploit-framework/lib/rex/ui/text/input/readline.rb:162:in `readline_with_output'
    from metasploit-framework/lib/rex/ui/text/input/readline.rb:100:in `pgets'
    from metasploit-framework/lib/rex/ui/text/shell.rb:188:in `run'
    from metasploit-framework/lib/metasploit/framework/command/console.rb:48:in `start'
    from metasploit-framework/lib/metasploit/framework/command/base.rb:82:in `start'
    from ./msfconsole:48:in `<main>'

brew cask install metasploit (installing the lastest version) and msfconsole -L shows

[-] * WARNING: Unable to load readline: cannot load such file -- readline
[-] * Falling back to RbReadLine
[-] ***

but starting ./msfconsole -L works fine is it possible to drop the readline dependency all together?

busterb commented 7 years ago

homebrew packages metasploit now? weird.

If you drop readline, it kills history support

busterb commented 7 years ago

please don't comment on 3 year old closed issues though - make a new one

busterb commented 7 years ago

Also, works fine here using the packages built by Rapid7:

msf auxiliary(http_login) > set pass_file /opt/metasploit-framework
metasploit-framework         metasploit-framework-extras  http_default_pass.txt

See https://github.com/rapid7/metasploit-framework/wiki/Nightly-Installers

busterb commented 7 years ago

Huh, homebrew packages the Rapid7 packages. Double weird!

==> Downloading https://osx.metasploit.com/metasploit-framework-4.15.0+201707031
########                                                                  11.9%

welp, I'll give it a shot.

busterb commented 7 years ago

hmm, still works fine for me. can you please open a fresh issue, and include a few things like the whole command history you used, and the os x version? Terminal version might be good to have too.