module tls give exception on error but don't write the result as in zgrab

[ghost]

I have done a large scan on my network for port 443, not all host which have open port 443 has a web server HTTPS, so zgrab2 give me an error with stack trace but don't log the event in output file. zgrab instead report all errors in output file, very useful for post-processing.

echo IP_WITH_443_OPEN_PORT_BUT_NO_HTTPS | /opt/scanner/sbin/zgrab2 tls --heartbleed --output-file=xxxxx.json
INFO[0000] started grab at 2018-03-26T09:39:30+02:00    
DEBU[0003] Failed to detect error from tls: oversized record received with length 20527 at goroutine 894 [running]:
runtime/debug.Stack(0xd9e8e0, 0xc420366d90, 0xc42003a040)
    /usr/lib/go-1.9/src/runtime/debug/stack.go:24 +0xa7
github.com/zmap/zgrab2.TryGetScanStatus(0xd9e8e0, 0xc420366d90, 0xc420366d90, 0xc420358e00)
    /root/go/src/github.com/zmap/zgrab2/status.go:92 +0x2c0
github.com/zmap/zgrab2/modules.(*TLSScanner).Scan(0xc42000e528, 0xc420368320, 0x10, 0x10, 0x0, 0x0, 0xa19060, 0x1, 0x4, 0x7f929d28acd0, ...)
    /root/go/src/github.com/zmap/zgrab2/modules/tls.go:72 +0xec
github.com/zmap/zgrab2.RunScanner(0xda9600, 0xc42000e528, 0xc420366d20, 0xc420368320, 0x10, 0x10, 0x0, 0x0, 0xa4a144, 0x3, ...)
    /root/go/src/github.com/zmap/zgrab2/scanner.go:32 +0xcc
github.com/zmap/zgrab2.grabTarget(0xc420368320, 0x10, 0x10, 0x0, 0x0, 0xc420366d20, 0x0, 0x0, 0x0)
    /root/go/src/github.com/zmap/zgrab2/processing.go:91 +0x245
github.com/zmap/zgrab2.Process.func2(0xc420057b00, 0xc420366d20, 0xc420057b60, 0xc4203682e0, 0x374)
        /root/go/src/github.com/zmap/zgrab2/processing.go:150 +0x143
created by github.com/zmap/zgrab2.Process
    /root/go/src/github.com/zmap/zgrab2/processing.go:143 +0x191

output file xxxx.json is empty with zgrab instead

echo IP_WITH_443_OPEN_PORT_BUT_NO_HTTPS | /opt/scanner/sbin/zgrab --port 443 --tls --heartbleed --output-file=xxxxx.json
Mar 26 09:39:50.173 [INFO] banner-grab: started grab at 2018-03-26T09:39:50+02:00
Mar 26 09:39:52.851 [ERROR] banner-grab: Conversation error with remote host 2.238.76.236:443: tls: oversized record received with length 20527
Mar 26 09:39:54.607 [INFO] banner-grab: finished grab (0 success; 1 failure) at 2018-03-26T09:39:54+02:00

output file xxxxx.json

cat xxxxx.json 
{"ip":"IP_WITH_443_OPEN_PORT_BUT_NO_HTTPS","timestamp":"2018-03-26T09:39:52+02:00","data":{"tls":{}},"error":"tls: oversized record received with length 20527","error_component":"tls"}

I have used golang 1.10 for tests.

[justinbastress]

Three tasks here:

1. Why is the host not showing up in the standard output?
2. Add detection for that TLS error (should be a SCAN_PROTOCOL_ERROR)
3. Change default log level back to INFO, not DEBU (and possibly lower the "Failed to detect error" messages' log level further, since this would only be used for internal debugging)

[justinbastress]

@etmatrix I can reproduce the error, except for the part about the output file being empty (see bottom), but since in your case I don't see the scan summary (e.g. [INFO] finished grab at 2018-04-04T10:06:12-04:00), I'm wondering if it isn't perhaps hung waiting on another host? (Currently zgrab2 has no timeout if one is not specified).

If you add --timeout=1 to your command, does that allow it to finish and populate xxxxx.json?

For what it's worth, we're adding a default timeout of 10s to zgrab2 (to match the existing behavior), but if that doesn't fix your problem, then we need to investigate that further.

Repro:

bash-3.1$ echo "HTTP/1.0 404 Not Found" | nc -l -p 443 > /dev/null &
[1] 6444
bash-3.1$ echo "127.0.0.1" | cmd/zgrab2/zgrab2 tls --heartbleed --port 443 --output-file=out.json
[INFO] started grab at 2018-04-04T10:06:12-04:00
[DEBU] Failed to detect error from tls: oversized record received with length 20527 at goroutine 28 [running]:
runtime/debug.Stack(0xda67a0, 0xc042cae000, 0xc042030040)
        /go/src/runtime/debug/stack.go:24 +0xae
github.com/zmap/zgrab2.TryGetScanStatus(0xda67a0, 0xc042cae000, 0xc042cae000, 0xc0423b2700)
        /go/src/github.com/zmap/zgrab2/status.go:92 +0x2c7
github.com/zmap/zgrab2/modules.(*TLSScanner).Scan(0xc0421023e0, 0xc042c96030, 0x10, 0x10, 0x0, 0x0, 0xc0423aa8c0, 0xc04203dca8, 0xc04242bbe0, 0x411b6d, ...)
        /go/src/github.com/zmap/zgrab2/modules/tls.go:72 +0xf3
github.com/zmap/zgrab2.RunScanner(0xdb10c0, 0xc0421023e0, 0xc0423ae780, 0xc042c96030, 0x10, 0x10, 0x0, 0x0, 0xa59911, 0x3, ...)
        /go/src/github.com/zmap/zgrab2/scanner.go:32 +0xd3
github.com/zmap/zgrab2.grabTarget(0xc042c96030, 0x10, 0x10, 0x0, 0x0, 0xc0423ae780, 0x0, 0x0, 0x0)
        /go/src/github.com/zmap/zgrab2/processing.go:92 +0x24c
github.com/zmap/zgrab2.Process.func2(0xc0421532c0, 0xc0423ae780, 0xc042153320, 0xc0422e3b20, 0x9)
        /go/src/github.com/zmap/zgrab2/processing.go:159 +0x14a
created by github.com/zmap/zgrab2.Process
        /go/src/github.com/zmap/zgrab2/processing.go:152 +0x198

[INFO] finished grab at 2018-04-04T10:06:12-04:00
{"statuses":{"tls":{"successes":0,"failures":1}},"start":"2018-04-04T10:06:12-04:00","end":"2018-04-04T10:06:12-04:00","duration":"14.9922ms"}
bash-3.1$ cat out.json
{"ip":"127.0.0.1","data":{"tls":{"status":"unknown-error","protocol":"tls","result":{"handshake_log":{},"heartbleed_log":{"heartbeat_enabled":false,"heartbleed_vulnerable":false}},"timestamp":"2018-04-04T10:06:12-04:00","error":"tls: oversized record received with length 20527"}}}
[1]+  Done                    echo "HTTP/1.0 404 Not Found" | nc -l -p 443 >/dev/null
bash-3.1$