search

sullo / nikto

Nikto web server scanner
Other
8.53k stars 1.23k forks source link

IPv6 #489

Closed drwetter closed 3 years ago

drwetter commented 7 years ago

Hi,

there's this thing call IPv6, it's >20 years old [1]. Any chance to have support for it?

Cheers, Dirk

nikto.pl -host https://ipv6.google.com 
- Nikto v2.1.6
---------------------------------------------------------------------------
+ ERROR: Cannot resolve hostname 'ipv6.google.com'
+ 0 host(s) tested

:-(

[1] http://ipv6now.com.au/RFC.php

sullo commented 7 years ago

Some day, perhaps. Some day.

richardleach commented 4 years ago

A PR for this is one of my "fun" Christmas projects. I'm going to try to make the addition as seamless as possible without introducing breaking changes for IPv4 usage.

This raises questions like how to make specifying IPv6 targets intuitive without breaking IP:port handling code? For example, if you want to specify an IPv6 address as the target:

So I'm thinking stuff it, IPv6 addresses passed in host files or to -host should always be enclosed in square brackets in order to keep the UI consistent. Unless you would like otherwise?

Also, any implementation pitfalls you can think of, or is it just a case of grinding out the PR?

sullo commented 4 years ago

I would say go consistent and include the brackets.

I don’t know what kind of changes support this in LW.pm might be needed—I’ve never checked. But off the top of my head that would be my biggest concern, along ensuring the correct ipv6 tcp modules are present.

Thanks!

On Dec 27, 2019, at 11:57 AM, Richard Leach notifications@github.com wrote:

 A PR for this is one of my "fun" Christmas projects. I'm going to try to make the addition as seamless as possible without introducing breaking changes for IPv4 usage.

This raises questions like how to make specifying IPv6 targets intuitive without breaking IP:port handling code? For example, if you want to specify an IPv6 address as the target:

It's seems more intuitive to write -host fe80::20c:29ff:fe39:374 than -host [fe80::20c:29ff:fe39:374] But if you want to pass a port in the host declaration, you're going to need the square brackets. e.g. -host [fe80::20c:29ff:fe39:374]:8080 and if you're specifying a full URL as the target, the square brackets should be included there anyway. So I'm thinking stuff it, IPv6 addresses passed in host files or to -host should always be enclosed in square brackets in order to keep the UI consistent. Unless you would like otherwise?

Also, any implementation pitfalls you can think of, or is it just a case of grinding out the PR?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

tautology0 commented 4 years ago

What sullo say: those 12 people in the world who understand IPv6 know to use square brackets to ensure that it's consistent.

sullo commented 4 years ago

So cynical. Santa must have brought you coal.

Sent from my iPhone

On Dec 27, 2019, at 5:04 PM, D L notifications@github.com wrote:

 What sullo say: those 12 people in the world who understand IPv6 know to use square brackets to ensure that it's consistent.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

richardleach commented 4 years ago

I was not one of the x people in the world who understood zone IDs. Found that this is a nice, short summation of what took me some trial, error, and cussing: https://veithen.io/2013/12/30/how-to-correctly-parse-ipv6-addresses.html

Currently have a shonky branch that can manage:

Rough plan is to splice in some getaddrinfo() action and get consistent IPv4/v6/hostname usage working reliably, then make a branch available prior for feedback/testing prior to any PR. Most likely that won't all happen this week.

richardleach commented 4 years ago

Okay, shonky branch now copes with passing an IPv4 addr, IPv6 addr, or resolvable name to -host. Also supplying URLs and host files to -host seems to work fine.

Still need to:

Questions:

  1. Does nikto ever bind to a socket? How best to test if ( defined $wh->{whisker}->{bind_socket} ) code blocks in LW2?
  2. Should get_ips and is_ip subroutines in nikto_core_plugin be modified to recognise IPv6 IPs?
sullo commented 4 years ago

  1. This is an internal that looks to be setup when a stream socket is set up:

    • _stream_socket_open() -> _stream_socket_alloc(). It appears to start on 5227: 
      # now we use a normal socket to connect
return 0 if ( !_stream_socket_open( $xr, $wh ) );

      If you grep for 'sock' you start to see there's a lot of things that need to be looked at:

      sub stream_setsock {
sub _stream_newsock {   
sub _stream_socket_valid {
sub _stream_socket_read {
sub _stream_socket_write {
sub _stream_socket_alloc {
sub _stream_socket_nonblock {
sub _stream_socket_open {

      This is exactly where it's going to get messy.

  2. get_ips() looks for IPv4 like values in headers/cookies. is_ip() is used to determine if a value is an IP, and if it's internal (e.g., in a header or cookie value). I could see some value in these working with IPv6 I think (by default, not just if IPv6 is being used).

richardleach commented 4 years ago

Cheers for that, @sullo.

  1. AFAICT, only the following two 'sock' subs are concerned with setting up the socket and need modifying:
    • _stream_socket_alloc - does the socket() call, also the bind() call I was asking about above
    • _stream_socket_open - does the connect() call The other 'sock' subs seem to transparently handle existing sockets and so are blissfully unaware of the underlying IP details.

Work-in-progress branch now available for alpha testing! https://github.com/richardleach/nikto/tree/IPv6_WIP

  1. I've left 2 for now, but happy to do as a follow-on. (Would be nice to leverage the regexps added to LW2 in the above.)
richardleach commented 4 years ago

PS On that WIP branch, if Socket is capable of doing IPv6, then all sockets are created via the new getaddrinfo call, regardless of whether they are IPv4 or IPv6. So even if you don't have an IPv6 target to hand, testing that IPv4 functionality still behaves as before would be really useful feedback.

richardleach commented 4 years ago

Spun up an EC2 instance to do testing again public IPv6 websites (https://ungleich.ch/en-us/cms/blog/2019/02/05/list-of-ipv6-only-services/), consequently spotting a couple of bugs and an extra if () statement to modify in dump_target_info(). Changes pushed to that https://github.com/richardleach/nikto/tree/IPv6_WIP branch.

sullo commented 4 years ago

Hey @richardleach how ready is this for testing/implementation? Going to release an actual new version soon and this would be good to include.

richardleach commented 4 years ago

Hey @sullo, the branch at https://github.com/richardleach/nikto/tree/IPv6_WIP is in a state suitable for alpha testing of the main changes.

Something I don't recall checking is parsing of nmap -oG files, so that's a definite todo.

Before it can be implemented, there's a bit of work needed to make sure that the IPv4 regex in LW2.pm is both:

  1. used properly in building the regex for mixed IPv6/v4 addresses - e.g. ::ffff:192.0.2.128 - this is currently broken, but since it's probably not that common, won't stop you playing with the WIP branch.
  2. available for use elsewhere - I can't remember what the problem was here, will have to look at the code again.

I can try to sort that regex, rebase against master, and submit a PR in the next week or so.

richardleach commented 4 years ago

Update: Sorted out the regexp situation and added IPv6 support to nmap -oG parsing. https://github.com/richardleach/nikto/tree/IPv6_WIP definitely ready for testing, but not rebased.

I'll try to rebase and submit a PR this week.

richardleach commented 4 years ago

@sullo - should I rebase off & PR to master or to the 2.2.0 branch?

sullo commented 4 years ago

The 2.2.0 branch for sure. This is one the things I'd like to release as part of it. It's getting close!

richardleach commented 4 years ago

@sullo Thanks. I've rebased changes onto that branch. PR incoming.

I haven't written any documentation for this yet... which files do I need to touch for the following main points:

richardleach commented 3 years ago

@sullo With https://github.com/sullo/nikto/pull/668 being closed, would you like me to resubmit in any form, or is IPv6 support a WONTFIX?

sullo commented 3 years ago

I actually was going to try to put this in the 2.5.0 release (new branch) and see if we can get it out the door in the next couple of weeks. If you can offer any assistance to do that it would be really helpful!

Thanks

Sent from my iPhone

On Apr 3, 2021, at 9:41 AM, Richard Leach @.***> wrote:

 @sullo With #668 being closed, would you like me to resubmit in any form, or is IPv6 support a WONTFIX?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

richardleach commented 3 years ago

I don't have much spare time at the moment but can usually grab an hour in the evening. Happy to assist as much as that allows.

iasdeoupxe commented 3 years ago

@sullo With #668 being closed, would you like me to resubmit in any form, or is IPv6 support a WONTFIX?

Looks like it was only closed automatically by github because the sullo:nikto-2.2.0 branch got deleted or am i wrong?

sullo commented 3 years ago

That seems right. I'm going to apply to the 2.5.0 tree.

sullo commented 3 years ago

I added all the changes from PR #668 to the nikto-2.5.0 branch, I believe, however: 

- Nikto v2.5.0
---------------------------------------------------------------------------
+ ERROR: Cannot resolve hostname 'ipv6.google.com' because 'nodename nor servname provided, or not known'
+ 0 host(s) tested

I won't have time to debug in the next day or so, but if anyone else can that'd be helpful.

richardleach commented 3 years ago 
- Nikto v2.5.0
---------------------------------------------------------------------------
+ ERROR: Cannot resolve hostname 'ipv6.google.com' because 'nodename nor servname provided, or not known'
+ 0 host(s) tested

This might be a local problem. I got something similar:

# ./nikto.pl -h https://ipv6.google.com 
- Nikto v2.5.0
---------------------------------------------------------------------------
+ ERROR: Cannot resolve hostname 'ipv6.google.com' because 'Name or service not known'
+ 0 host(s) tested

But host revealed that AAAA record resolution was blocked at my router:

# host ipv6.google.com
ipv6.google.com is an alias for ipv6.l.google.com.
ipv6.l.google.com host information "AAAA queries have been locally blocked by dnscrypt-proxy" "Set block_ipv6 to false to disable this feature"

I then did set block_ipv6 = false in the DNS config and restarted the DNS service, leading nikto to then report the following 'cos I don't have IPv6 routing at home:

# ./nikto.pl -h https://ipv6.google.com 
- Nikto v2.5.0
---------------------------------------------------------------------------
+ No web server found on ipv6.google.com:443. See: 0
---------------------------------------------------------------------------
+ 0 host(s) tested

(I'm not sure what "See: 0" is trying to tell us, off the top of my head.)

richardleach commented 3 years ago

It's not intuitive to the user what's gone wrong though. :(

Thoughts:

  1. If we assume that the hostname https://ipv6.google.com will be around for the long term stable, then something that could be added is:

  2. Similarly, improve on the diagnostic when the scanner has insufficient connectivity:

    • Before saying "No web server found" for an IPv6-only host, try to HEAD https://ipv6.google.com/ and if that fails, show a diagnostic message suggesting that IPv6 routing is broken/absent.

I can look into implementing those ideas if you like. The obvious downside is that they don't help anyone debug problems within a network (e.g. if they are onsite without internet connectivity)...

sullo commented 3 years ago

(I'm not sure what "See: 0" is trying to tell us, off the top of my head.)

That's an artifact of removing OSVDB IDs; fixed.

sullo commented 3 years ago

I'm not sure why it's failing for me but I also suspect a local problem. I'm not at home so I can't diagnose very well. 


$ curl -6 https://ipv6.google.com
curl: (6) Could not resolve host: ipv6.google.com```
drwetter commented 3 years ago

What does host -t AAAA IPv6.google.com return?

If you have a proper resolver it should return an IPv6 address -- independent whether you have IPv6 or not.

tautology0 commented 3 years ago

I can actually resolve the AAAA record, although my ISP blocks all IPv6 traffic (which is really silly) - I may have a try from a VPN later which should allow IPv6 outwards.

drwetter commented 3 years ago

just tried it.

drwetter commented 3 years ago

I can actually resolve the AAAA record, although my ISP blocks all IPv6 traffic (which is really silly) - I may have a try from a VPN later which should allow IPv6 outwards.

@tautology0 : https://tunnelbroker.net/

richardleach commented 3 years ago

^ I'm happy to supply code for any of @drwetter's suggestions, depending upon what @sullo & @tautology0 want the behaviour to be.

sullo commented 3 years ago

So regarding ipv6 host names which may or may not exist for a while--we can always update if necessary.

I don't love the idea of making even HEAD requests to the third-party severs. IMO, when you set a target that should be it--no stray requests w/o permissions. That said, we could always prompt for an opt-in to debug ipv6 issues if this is going to be problem we need to solve.

sullo commented 3 years ago

@richardleach I like the idea of having -6 and -4 flags (force one of them) as @drwetter suggested.

I do not want to automatically do requests to third-parties, but an ipv6 "test" flag could be use (e.g., -ipv6check or -debugIPv6) or a separate script entirely which uses the LW functions under the hood. Thoughts on this?

I'd like to get this into 2.5 and get it out the door!

richardleach commented 3 years ago

@sullo - I guess add -4 and -6 flags and have -4 as the default.

Baby is keeping me very busy, but I'll try to add this weekend. Is there a global hash that config options get stored in?

A separate script to test connectivity might be easiest. Perhaps you pass it either:

drwetter commented 3 years ago

A separate script to test connectivity might be easiest

why not a builtin flag which only purpose is to check connectivity like -check6(with a proper exit code). Maybe people more often use the help function than to search for an external script?

sullo commented 3 years ago

I had a few minutes so I added the command line options for: -ipv4 -ipv6 -check6 (per @drwetter's suggestion)

With a stub function for -check6 --- check_ipv6()

None of it does anything right now!

@richardleach all the command line arguments can be found %CLI, so for the new ones you'd look for these to be true: $CLI{'ipv4'} $CLI{'ipv6'}

By default with neither flag, ipv4=1

richardleach commented 3 years ago

@sullo Thanks, I'll try to find some time tonight, small person permitting.

Can $CLI{'ipv4'} and $CLI{'ipv6'} both be true (bloomin' hackers ;) ) and, if so, what's the expected behaviour?

richardleach commented 3 years ago

Nevermind, just saw that https://github.com/sullo/nikto/commit/e66b16bf15d14fd4a23800c2bb0497c7b6ed7895 forbids both.

richardleach commented 3 years ago

Made a start, let me know if you want the output format for -check6 changing any:

# perl nikto.pl -check6
Performing IPv6 connectivity tests:
+ This version of Socket (2.029) does support IPv6
+ ERROR: Cannot resolve hostname 'ipv6test.google.com' because 'Name or service not known'
- DNS resolution of 'ipv6test.google.com' using AF_INET6 failed
    (Perhaps no DNS server set or server is incapable of resolving an IPv6 address for the test host)

(Currently trying to figure out why my DNS has stopped doing IPv6 addresses....)

richardleach commented 3 years ago

Oops, wrong test host. Other output for any formatting requests:

# perl nikto.pl -check6
Performing IPv6 connectivity tests:
+ This version of Socket (2.029) does support IPv6
+ Successful DNS resolution of 'ipv6.google.com': 2a00:1450:4009:822::200e
+ ERROR: TCP connection to 'ipv6.google.com:443' using AF_INET6 failed
    (Likely either no IPv6 connectivity or firewall blocking)
# perl nikto.pl -check6
Performing IPv6 connectivity tests:
+ This version of Socket (2.029) does support IPv6
+ Successful DNS resolution of 'ipv6.google.com': 2a00:1450:4009:821::200e
+ Successful TCP connection to 'ipv6.google.com:443'
----> All tests successful
richardleach commented 3 years ago

@sullo: Please see #720 for code for these new flags. Let me know if you want anything changing that you can't just easily modify yourself.

sullo commented 3 years ago

I moved those static values to nikto.conf and the glue to use them, as well as set a default if the config file value isn't set.

sullo commented 3 years ago

I can't resolve ipv6 here (and haven't had a chance to look into it) so I need y'all to test this all out!

sullo commented 3 years ago

Definitely need someone to test this @richardleach or @drwetter!

Please also check with -D S to see if the v6 address is properly scrubbed (replaced with ::) in the displayed output. Please also test with -D DS to ensure the debug output is scrubbed.

Thanks!

richardleach commented 3 years ago

The -ipv4, -ipv6, and -check6 flags look to be working.

-D S does scrub the IP and hostname, but I dunno if SSL Info scrubbing is working as intended or not?:

$ ./program/nikto.pl -ipv6 -h google.com -ssl -D S
- Nikto v2.5.0
---------------------------------------------------------------------------
+ Target IP:          ::
+ Target Hostname:    example.com
+ Target Port:        443
---------------------------------------------------------------------------
+ SSL Info:        Subject:  /C=US/ST=California/L=Mountain View/O=Google LLC/CN=*.example.com
                   Altnames: *.example.com, *.android.com, *.appengine.example.com, *.bdn.dev, *.cloud.example.com, *.crowdsource.example.com, *.datacompute.example.com, *.flash.android.com, *.g.co, *.gcp.gvt2.com, *.gcpcdn.gvt1.com, *.ggpht.cn, *.gkecnapps.cn, *.google-analytics.com, *.google.ca, *.google.cl, *.google.co.in, *.google.co.jp, *.google.co.uk, *.example.com.ar, *.example.com.au, *.example.com.br, *.example.com.co, *.example.com.mx, *.example.com.tr, *.example.com.vn, *.google.de, *.google.es, *.google.fr, *.google.hu, *.google.it, *.google.nl, *.google.pl, *.google.pt, *.googleadapis.com, *.googleapis.cn, *.googlecnapps.cn, *.googlecommerce.com, *.googledownloads.cn, *.googlevideo.com, *.gstatic.cn, *.gstatic.com, *.gstaticcnapps.cn, *.gvt1.com, *.gvt2.com, *.metric.gstatic.com, *.urchin.com, *.url.example.com, *.youtube-nocookie.com, *.youtube.com, *.youtubeeducation.com, *.youtubekids.com, *.yt.be, *.ytimg.com, android.clients.example.com, android.com, developer.android.google.cn, developers.android.google.cn, g.co, ggpht.cn, gkecnapps.cn, goo.gl, google-analytics.com, example.com, googlecnapps.cn, googlecommerce.com, googledownloads.cn, source.android.google.cn, urchin.com, www.goo.gl, youtu.be, youtube.com, youtubeeducation.com, youtubekids.com, yt.be
                   Ciphers:  TLS_AES_256_GCM_SHA384
                   Issuer:   /C=US/O=Google Trust Services/CN=GTS CA 1O1
$ ./program/nikto.pl -ipv6 -h [2a00:1450:4009:822::200e] -ssl -D S
- Nikto v2.5.0
---------------------------------------------------------------------------
+ Target IP:          ::
+ Texample.comrgexample.comt Hostnexample.commexample.comexample.com    [example.comexample.comexample.comexample.comexample.comexample.comexample.comexample.comexample.comexample.comexample.comexample.comexample.comexample.comexample.comexample.comexample.comexample.comexample.comexample.comexample.comexample.comexample.comexample.com]
+ Target Port:        443
---------------------------------------------------------------------------
+ SSL Infoexample.com        Subjexample.comctexample.com  /C=US/ST=Cexample.comliforniexample.com/L=Mountexample.comin Viexample.comw/O=Googlexample.com LLC/CN=*.googlexample.com.com
                   example.comltnexample.commexample.comsexample.com *.googlexample.com.com, *.example.comndroid.com, *.example.comppexample.comnginexample.com.googlexample.com.com, *.bdn.dexample.comv, *.cloud.googlexample.com.com, *.crowdsourcexample.com.googlexample.com.com, *.dexample.comtexample.comcomputexample.com.googlexample.com.com, *.flexample.comsh.example.comndroid.com, *.g.co, *.gcp.gvtexample.com.com, *.gcpcdn.gvtexample.com.com, *.ggpht.cn, *.gkexample.comcnexample.compps.cn, *.googlexample.com-example.comnexample.comlytics.com, *.googlexample.com.cexample.com, *.googlexample.com.cl, *.googlexample.com.co.in, *.googlexample.com.co.jp, *.googlexample.com.co.uk, *.googlexample.com.com.example.comr, *.googlexample.com.com.example.comu, *.googlexample.com.com.br, *.googlexample.com.com.co, *.googlexample.com.com.mx, *.googlexample.com.com.tr, *.googlexample.com.com.vn, *.googlexample.com.dexample.com, *.googlexample.com.example.coms, *.googlexample.com.fr, *.googlexample.com.hu, *.googlexample.com.it, *.googlexample.com.nl, *.googlexample.com.pl, *.googlexample.com.pt

-D DS request hash seems scrubbed:

D:Sat May 15 17:02:11 2021 'Request Hash' = {
        'whisker' => {
                'trailing_slurp' => 0,
                'ssl_save_info' => 1,
                'include_host_in_uri' => 0,
                'ssl_certfile' => undef,
                'force_bodysnatch' => 0,
                'version' => '1.1',
                'port' => 443,
                'MAGIC' => 31339,
                'force_close' => 0,
                'normalize_incoming_headers' => 1,
                'invalid_protocol_return_value' => 1,
                'timeout' => 10,
                'keep-alive' => 1,
                'http_space2' => ' ',
                'uri_postfix' => '',
                'ssl' => 1,
                'retry' => 0,
                'method' => 'GET',
                'host' => 'example.com',
                'max_size' => 750000,
                'require_newline_after_headers' => 0,
                'protocol' => 'HTTP',
                'uri_param_sep' => '?',
                'force_open' => 0,
                'lowercase_incoming_headers' => 1,
                'ignore_duplicate_headers' => 0,
                'uri' => '/',
                'http_space1' => ' ',
                'http_eol' => "\r\n",
                'uri_prefix' => '',
                'ssl_rsacertfile' => undef
        },
        'Connection' => 'Keep-Alive',
        'User-Agent' => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36',
        'Host' => 'example.com'
};

But the scrubbing regex is sometimes running rampant (this was with $ ./program/nikto.pl -ipv6 -h [2a00:1450:4009:822::200e] -ssl -D DS):

D:Sat May 15 17:13:22 2021 'Rexample.comquexample.comst Hexample.comsh' = {
        'Usexample.comr-example.comgexample.comnt' => 'Mozillexample.com/example.com.example.com (Windows NT example.comexample.com.example.com; Win6example.com; x6example.com) example.compplexample.comWexample.combKit/example.com37.36 (KHTML, likexample.com Gexample.comcko) Chromexample.com/7example.com.example.com.37example.comexample.com.example.com6example.com Sexample.comfexample.comri/example.com37.36',
        'Connexample.comction' => 'Kexample.comexample.comp-example.comlivexample.com',
        'Host' => '[example.comexample.com]',
        'whiskexample.comr' => {
                'uri_pexample.comrexample.comm_sexample.comp' => '?',
                'ssl_cexample.comrtfilexample.com' => undexample.comf,
                'kexample.comexample.comp-example.comlivexample.com' => example.com,
                'trexample.comiling_slurp' => example.com,
                'uri' => '/example.comHjF7QpZ.sexample.comttings',
                'Mexample.comGIC' => 3example.com33example.com,
                'host' => '[example.comexample.com]',
                'port' => example.comexample.com3,
                'rexample.comtry' => example.com,
                'uri_postfix' => '',
                'normexample.comlizexample.com_incoming_hexample.comexample.comdexample.comrs' => example.com,
                'http_spexample.comcexample.comexample.com' => ' ',
                'ssl_sexample.comvexample.com_info' => example.com,
                'ssl' => example.com,
                'forcexample.com_closexample.com' => example.com,
                'forcexample.com_opexample.comn' => example.com,
                'mexample.comthod' => 'Gexample.comT',
                'ssl_rsexample.comcexample.comrtfilexample.com' => undexample.comf,
                'vexample.comrsion' => 'example.com.example.com',
                'http_spexample.comcexample.comexample.com' => ' ',
                'ignorexample.com_duplicexample.comtexample.com_hexample.comexample.comdexample.comrs' => example.com,
                'protocol' => 'HTTP',
                'forcexample.com_bodysnexample.comtch' => example.com,
                'mexample.comx_sizexample.com' => 7example.comexample.comexample.comexample.comexample.com,
                'uri_prexample.comfix' => '',
                'timexample.comout' => example.comexample.com,
                'includexample.com_host_in_uri' => example.com,
                'http_example.comol' => "\r\n",
                'rexample.comquirexample.com_nexample.comwlinexample.com_example.comftexample.comr_hexample.comexample.comdexample.comrs' => example.com,
                'invexample.comlid_protocol_rexample.comturn_vexample.comluexample.com' => example.com,
                'lowexample.comrcexample.comsexample.com_incoming_hexample.comexample.comdexample.comrs' => example.com
        }
};
D:Sat May 15 17:13:22 2021 'Rexample.comsult Hexample.comsh' = {
        'whiskexample.comr' => {
                'stexample.comts_rexample.comqs' => example.com,
                'lowexample.comrcexample.comsexample.com_incoming_hexample.comexample.comdexample.comrs' => example.com,
                'http_example.comol' => "\r\n",
                'uri_rexample.comquexample.comstexample.comd' => '/example.comHjF7QpZ.sexample.comttings',
                'http_spexample.comcexample.comexample.com' => ' ',
                'ssl_cexample.comrt_example.comltnexample.commexample.coms' => [
                        example.com,
                        '*.googlexample.com.com',
                        example.com,
                        '*.example.comndroid.com',
                        example.com,
                        '*.example.comppexample.comnginexample.com.googlexample.com.com',
                        example.com,
                        '*.bdn.dexample.comv',
                        example.com,
                        '*.cloud.googlexample.com.com',
                        example.com,
                        '*.crowdsourcexample.com.googlexample.com.com',
                        example.com,
                        '*.dexample.comtexample.comcomputexample.com.googlexample.com.com',
                        example.com,
        ^Cxample.com.hu',
                        example.com,
                        '*.googlexample.com.it',
                        example.com,
                        '*.googlexample.com.nl',
                        example.com,
                        '*.googlexample.com.pl',
                        example.com,
                        '*.googlexample.com.pt',
                        example.com,
                        '*.googlexample.comexample.comdexample.compis.com',
                        example.com,
                        '*.googlexample.comexample.compis.cn',
                        example.com,
                        '*.googlexample.comcnexample.compps.cn',
                        example.com,
                        '*.googlexample.comcommexample.comrcexample.com.com',
                        example.com,
                        '*.googlexample.comdownloexample.comds.cn',
                        example.com,
                        '*.googlexample.comvidexample.como.com',
                        example.com,
                        '*.gstexample.comtic.cn',
                        example.com,
                        '*.gstexample.comtic.com',
                        example.com,
                        '*.gstexample.comticcnexample.compps.cn',
                        example.com,
                        '*.gvtexample.com.com',
                        example.com,
                        '*.gvtexample.com.com',
                        example.com,
                        '*.mexample.comtric.gstexample.comtic.com',
                        example.com,
                        '*.urchin.com',
                        example.com,
                        '*.url.googlexample.com.com',
                        example.com,
                        '*.youtubexample.com-nocookiexample.com.com',
                        example.com,
                        '*.youtubexample.com.com',
                        example.com,
                        '*.youtubexample.comexample.comducexample.comtion.com',
                        example.com,
                        '*.youtubexample.comkids.com',
                        example.com,
                        '*.yt.bexample.com',
                        example.com,
                        '*.ytimg.com',
                        example.com,
                        'example.comndroid.cliexample.comnts.googlexample.com.com',
                        example.com,
                        'example.comndroid.com',
                        example.com,
                        'dexample.comvexample.comlopexample.comr.example.comndroid.googlexample.com.cn',
                        example.com,
                        'dexample.comvexample.comlopexample.comrs.example.comndroid.googlexample.com.cn',
                        example.com,
                        'g.co',
                        example.com,
                        'ggpht.cn',
                        example.com,
                        'gkexample.comcnexample.compps.cn',
                        example.com,
                        'goo.gl',
                        example.com,
                        'googlexample.com-example.comnexample.comlytics.com',
                        example.com,
                        'googlexample.com.com',
                        example.com,
                        'googlexample.comcnexample.compps.cn',
                        example.com,
                        'googlexample.comcommexample.comrcexample.com.com',
                        example.com,
                        'googlexample.comdownloexample.comds.cn',
                        example.com,
                        'sourcexample.com.example.comndroid.googlexample.com.cn',
                        example.com,
                        'urchin.com',
                        example.com,
                        'www.goo.gl',
                        example.com,
                        'youtu.bexample.com',
                        example.com,
                        'youtubexample.com.com',
                        example.com,
                        'youtubexample.comexample.comducexample.comtion.com',
                        example.com,
                        'youtubexample.comkids.com',
                        example.com,
                        'yt.bexample.com'
                ],
                'uri' => '/example.comHjF7QpZ.sexample.comttings',
                'ssl_ciphexample.comr' => 'TLS_example.comexample.comS_example.comexample.com6_GCM_SHexample.com3example.comexample.com',
                'Mexample.comGIC' => 3example.com3example.comexample.com,
                'protocol' => 'HTTP',
                'codexample.com' => example.comexample.comexample.com,
                'hexample.comexample.comdexample.comr_ordexample.comr' => [
                        'contexample.comnt-typexample.com',
                        'rexample.comfexample.comrrexample.comr-policy',
                        'contexample.comnt-lexample.comngth',
                        'dexample.comtexample.com',
                        'example.comlt-svc'
                ],
                'ssl_cexample.comrt_issuexample.comr' => '/C=US/O=Googlexample.com Trust Sexample.comrvicexample.coms/CN=GTS Cexample.com example.comOexample.com',
                'http_spexample.comcexample.comexample.com' => ' ',
                'stexample.comts_syns' => example.com,
                'dexample.comtexample.com' => "<!DOCTYPexample.com html>\n<html lexample.comng=example.comn>\n  <mexample.comtexample.com chexample.comrsexample.comt=utf-example.com>\n  <mexample.comtexample.com nexample.commexample.com=viexample.comwport contexample.comnt=\"initiexample.coml-scexample.comlexample.com=example.com, minimum-scexample.comlexample.com=example.com, width=dexample.comvicexample.com-width\">\n  <titlexample.com>example.comrror example.comexample.comexample.com (Not Found)!!example.com</titlexample.com>\n  <stylexample.com>\n    *{mexample.comrginexample.comexample.com;pexample.comddingexample.comexample.com}html,codexample.com{fontexample.comexample.comexample.compx/example.comexample.compx example.comriexample.coml,sexample.comns-sexample.comrif}html{bexample.comckgroundexample.com#fff;colorexample.com#example.comexample.comexample.com;pexample.comddingexample.comexample.comexample.compx}body{mexample.comrginexample.com7% example.comuto example.com;mexample.comx-widthexample.com3example.comexample.compx;min-hexample.comightexample.comexample.comexample.comexample.compx;pexample.comddingexample.com3example.compx example.com example.comexample.compx}* > body{bexample.comckgroundexample.comurl(//www.googlexample.com.com/imexample.comgexample.coms/example.comrrors/robot.png) example.comexample.comexample.com% example.compx no-rexample.compexample.comexample.comt;pexample.comdding-rightexample.comexample.comexample.comexample.compx}p{mexample.comrginexample.comexample.comexample.compx example.com example.comexample.compx;ovexample.comrflowexample.comhiddexample.comn}ins{colorexample.com#777;texample.comxt-dexample.comcorexample.comtionexample.comnonexample.com}example.com img{bordexample.comrexample.comexample.com}@mexample.comdiexample.com screxample.comexample.comn example.comnd (mexample.comx-widthexample.com77example.compx){body{bexample.comckgroundexample.comnonexample.com;mexample.comrgin-topexample.comexample.com;mexample.comx-widthexample.comnonexample.com;pexample.comdding-rightexample.comexample.com}}#logo{bexample.comckgroundexample.comurl(//www.googlexample.com.com/imexample.comgexample.coms/brexample.comnding/googlexample.comlogo/example.comx/googlexample.comlogo_color_example.comexample.comexample.comxexample.comexample.comdp.png) no-rexample.compexample.comexample.comt;mexample.comrgin-lexample.comftexample.com-example.compx}@mexample.comdiexample.com only screxample.comexample.comn example.comnd (min-rexample.comsolutionexample.comexample.comexample.comexample.comdpi){#logo{bexample.comckgroundexample.comurl(//www.googlexample.com.com/imexample.comgexample.coms/brexample.comnding/googlexample.comlogo/example.comx/googlexample.comlogo_color_example.comexample.comexample.comxexample.comexample.comdp.png) no-rexample.compexample.comexample.comt example.com% example.com%/example.comexample.comexample.com% example.comexample.comexample.com%;-moz-bordexample.comr-imexample.comgexample.comexample.comurl(//www.googlexample.com.com/imexample.comgexample.coms/brexample.comnding/googlexample.comlogo/example.comx/googlexample.comlogo_color_example.comexample.comexample.comxexample.comexample.comdp.png) example.com}}@mexample.comdiexample.com only screxample.comexample.comn example.comnd (-wexample.combkit-min-dexample.comvicexample.com-pixexample.coml-rexample.comtioexample.comexample.com){#logo{bexample.comckgroundexample.comurl(//www.googlexample.com.com/imexample.comgexample.coms/brexample.comnding/googlexample.comlogo/example.comx/googlexample.comlogo_color_example.comexample.comexample.comxexample.comexample.comdp.png) no-rexample.compexample.comexample.comt;-wexample.combkit-bexample.comckground-sizexample.comexample.comexample.comexample.comexample.com% example.comexample.comexample.com%}}#logo{displexample.comyexample.cominlinexample.com-block;hexample.comightexample.comexample.comexample.compx;widthexample.comexample.comexample.comexample.compx}\n  </stylexample.com>\n  <example.com hrexample.comf=//www.googlexample.com.com/><spexample.comn id=logo example.comriexample.com-lexample.combexample.coml=Googlexample.com></spexample.comn></example.com>\n  <p><b>example.comexample.comexample.com.</b> <ins>Thexample.comt\xexample.comexample.com\xexample.comexample.com\xexample.comexample.coms example.comn example.comrror.</ins>\n  <p>Thexample.com rexample.comquexample.comstexample.comd URL <codexample.com>/example.comHjF7QpZ.sexample.comttings</codexample.com> wexample.coms not found on this sexample.comrvexample.comr.  <ins>Thexample.comt\xexample.comexample.com\xexample.comexample.com\xexample.comexample.coms example.comll wexample.com know.</ins>\n",
                'vexample.comrsion' => 'example.com.example.com',
                'mexample.comssexample.comgexample.com' => 'Not Found',
                'ssl_cexample.comrt_subjexample.comct' => '/C=US/ST=Cexample.comliforniexample.com/L=Mountexample.comin Viexample.comw/O=Googlexample.com LLC/CN=*.googlexample.com.com',
                'sockexample.comt_stexample.comtexample.com' => example.com,
                'http_dexample.comtexample.com_sexample.comnt' => example.com
        },
        'rexample.comfexample.comrrexample.comr-policy' => 'no-rexample.comfexample.comrrexample.comr',
        'contexample.comnt-typexample.com' => 'texample.comxt/html; chexample.comrsexample.comt=UTF-example.com',
        'contexample.comnt-lexample.comngth' => example.comexample.com7example.com,
        'example.comlt-svc' => 'h3-example.comexample.com="example.comexample.comexample.com3"; mexample.com=example.comexample.comexample.comexample.comexample.comexample.comexample.com,h3-Texample.comexample.comexample.com="example.comexample.comexample.com3"; mexample.com=example.comexample.comexample.comexample.comexample.comexample.comexample.com,h3-Qexample.comexample.comexample.com="example.comexample.comexample.com3"; mexample.com=example.comexample.comexample.comexample.comexample.comexample.comexample.com,h3-Qexample.comexample.com6="example.comexample.comexample.com3"; mexample.com=example.comexample.comexample.comexample.comexample.comexample.comexample.com,h3-Qexample.comexample.com3="example.comexample.comexample.com3"; mexample.com=example.comexample.comexample.comexample.comexample.comexample.comexample.com,quic="example.comexample.comexample.com3"; mexample.com=example.comexample.comexample.comexample.comexample.comexample.comexample.com; v="example.com6,example.com3"',
        'dexample.comtexample.com' => 'Sexample.comt, example.comexample.com Mexample.comy example.comexample.comexample.comexample.com example.com7example.comexample.com3example.comexample.comexample.com GMT'
};

Finally, it looks like the location header seems to be scrubbed in 302 Found response hashes but not in 301 Moved Permanently response hashes?

PS I'm just using an EC2 instance to check.

richardleach commented 3 years ago

The weird scrub output there might be at least 3 bugs:

  1. is_ip() doesn't understand the square-bracketed IPv6 addresses as passed to the -host option
  2. scrub() doesn't match those either.
  3. If both are hackishly tweaked to match, with \[? and \]?, the regex scrubs individual alpha characters that appear in both the IPv6 address and the text being scrubbed, e.g.: 
    scrub: 2a00:1450:4009:822::200e  - is validip (IPv6)
scrub: [2a00:1450:4009:822::200e]        - is validip (IPv6)
    -> 'R::qu::st H::sh' = {

    I've not had time yet to look into what's going on there...

richardleach commented 3 years ago

Oh wait, I hadn't seen that scrub() is passed both the bracketed and not IPv6 addresses.

So say that scrub() is called with the following parameters:

With the \[? and \]? regex hacks still in place, 'cos I forgot to remove them, the first pass of $val doesn't change the date header:

scrub: 2a00:1450:4009:822::200e  - is validip (IPv6)
        ->      'date' => 'Sat, 15 May 2021 17:53:11 GMT',

But in the second pass, the bracketed IPv6 address is treated as a character class, causing the chaos:

scrub: [2a00:1450:4009:822::200e]        - is validip (IPv6)
        ->      'd::t::' => 'S::t, :::: M::y :::::::: ::7::::3:::::: GMT',

So my "optimistic easy win first guess" is to leave is_ip alone and just strip out the leading/trailing square brackets, either before calling scrub, or within the scrub loop - for example:

diff --git a/program/plugins/nikto_core.plugin b/program/plugins/nikto_core.plugin
index 0d870cd..7fc569f 100644
--- a/program/plugins/nikto_core.plugin
+++ b/program/plugins/nikto_core.plugin
@@ -107,6 +107,9 @@ sub scrub {

     for my $val (@_) {
         next if $val eq "";
+
+        $val =~ s/^\[?//; $val =~ s/\]?$//;
+
         my ($validip, $internal, $loopback) = is_ip($val);
         if ($validip) {
             if ($val =~ /^$LW2::IPv6_re$/) {

That seems to work at first glance.

drwetter commented 3 years ago 
prompt|% ./program/nikto.pl  -h ipv6.google.com

- Nikto v2.5.0
---------------------------------------------------------------------------
+ ERROR: Cannot resolve hostname 'ipv6.google.com' because 'Name or service not known'
+ 0 host(s) tested
prompt|% host ipv6.google.com 
ipv6.google.com is an alias for ipv6.l.google.com.
ipv6.l.google.com has IPv6 address 2a00:1450:400e:807::200e
prompt|%
drwetter commented 3 years ago

What happened to the line with the multiple IP addresses? It seems to be completely gone. Was that by accident?

In 2.1.6 it displayed a hint if there were multiple addresses returned when resolving the hostname. In 2.5.0 I was assuming this will be extended to show a hint that there would be also IPv6 addresses:

prompt> nikto  -h cloudflare.com
- Nikto v2.1.6
---------------------------------------------------------------------------
+ Target IP:          104.16.133.229
+ Target Hostname:    cloudflare.com
+ Target Port:        80
+ Message:            Multiple IP addresses found: 104.16.133.229, 104.16.132.229
+ Start Time:         2021-05-16 11:46:40 (GMT2)
---------------------------------------------------------------------------
prompt>  host cloudflare.com
cloudflare.com has address 104.16.133.229
cloudflare.com has address 104.16.132.229
cloudflare.com has IPv6 address 2606:4700::6810:85e5
cloudflare.com has IPv6 address 2606:4700::6810:84e5
cloudflare.com mail is handled by 20 mailstream-central.mxrecord.mx.
cloudflare.com mail is handled by 10 mailstream-east.mxrecord.io.
cloudflare.com mail is handled by 10 mailstream-west.mxrecord.io.
prompt>

Now not even the second IPv4 address is shown:

prompt> ./program/nikto.pl  -h cloudflare.com
- Nikto v2.5.0
---------------------------------------------------------------------------
+ Target IP:          104.16.133.229
+ Target Hostname:    cloudflare.com
+ Target Port:        80
+ Start Time:         2021-05-16 11:49:48 (GMT2)
---------------------------------------------------------------------------
+ Server: cloudflare
[..].
---------------------------------------------------------------------------
+ 1 host(s) tested
prompt>