w3af 1.6 does not run correclty on Kali

[marrowsmith]

Created a new working directory, verified all dependencies are installed, ui and console both launch but are unable to create a scan using the web_spider plugin.

Using the default w3af 1.2 version shipped with kali, I am successful in starting a new scan using the web_spider plugin

[andresriancho]

Can't really debug anything with this amount of detail. What error(s) are you seeing?

[marrowsmith]

What debug info do you need?

www server is running and can successfully launch scans/attacks via w3af 1.2

[marrowsmith]

This is my output from my scan using 1.6

w3af>>> start Called w3afCore.start() Enabling _dns_cache() Called build_openers DNS response from DNS server for domain: xxxx.com Exception while inserting request/response to the database: Expected dict, got Headers The request/response that generated the error is: 1 xxxxx.com/ 303

IMPORTANT The following error was detected by w3af and couldn't be resolved: The remote web server is not answering our HTTP requests, multiple errors have been found while trying to GET a response from the server.

In most cases this means that the configured target is incorrect, the port is closed, there is a firewall blocking our packets or there is no HTTP daemon listening on that port.

Please verify your target configuration and try again.

Scan finished in 0 seconds. w3af>>> exit w3af>>> The user stopped the core, finishing threads... 0 seconds. were needed to stop the core.

GPL inside.

root@xxxx:~/Desktop/work/w3af# wget https://xxxx.com/ --2013-10-01 14:58:59-- https://xxxxx.com/ Resolving xxxxxx.com (xxxxxx.com)... x.x.x.x Connecting to xxxxxx.com (xxxxxx.com)|x.x.x.x|:443... connected. HTTP request sent, awaiting response... 303 See Other Location: https://xxxxx.com/xxxx/ [following] --2013-10-01 14:59:00-- https://xxxxxx.com/xxxx/ Reusing existing connection to xxxxxx.com:443. HTTP request sent, awaiting response... 303 See Other Location: https://xxxxxx.com/xxxxx [following] --2013-10-01 14:59:00-- https://xxxxxx.com/xxxxxx Reusing existing connection to xxxxxx.com:443. HTTP request sent, awaiting response... 200 OK Length: 8326 (8.1K) [text/html] Saving to: `index.html'

100%[======================================>] 8,326 52.2K/s in 0.2s

2013-10-01 14:59:00 (52.2 KB/s) - `index.html' saved [8326/8326]

[ghost]

i have the same problem :( please help me

[andresriancho]

This seems to be the problem: "Exception while inserting request/response to the database: Expected dict, got Headers", anyone wants to grep for that in the code and debug it?

[marrowsmith]

core/data/url/handlers/cache_backend/db.py

@staticmethod def store_in_cache(request, response):

Create the http response object

    resp = HTTPResponse.from_httplib_resp(response,
                                          original_url=request.url_object)
    resp.set_id(response.id)
    resp.set_alias(gen_hash(request))

    hi = HistoryItem()
    hi.request = request
    hi.response = resp

    # Now save them
    try:
        hi.save()
    except sqlite3.Error, e:
        msg = 'A sqlite3 error was raised: "%s".' % e

        if 'disk' in str(e).lower():
            msg += ' Please check if your disk is full.'

        raise w3afMustStopException(msg)

    except Exception, ex:
        msg = ('Exception while inserting request/response to the'
               ' database: %s\nThe request/response that generated'
               ' the error is: %s %s %s' %
               (ex, resp.get_id(), request.get_uri(), resp.get_code()))
        om.out.error(msg)
        raise Exception(msg)

still combing through the files.

[marrowsmith]

am a little lost on this now, not finding anything obvious that is wrong.

[andresriancho]

Well, this seems to be a rather common issue

[andresriancho]

Find this code in cache_backend/db.py:

        except Exception, ex:
            msg = ('Exception while inserting request/response to the'
                   ' database: %s\nThe request/response that generated'
                   ' the error is: %s %s %s' %
                   (ex, resp.get_id(), request.get_uri(), resp.get_code()))
            om.out.error(msg)
            raise Exception(msg)

Replace the raise Exception(msg) with raise and run the same scan again. Please send me the new detailed information.

[marrowsmith]

DNS response from DNS server for domain: XXXX.com Exception while inserting request/response to the database: Expected dict, got Headers The request/response that generated the error is: 1 https://XXXX.com/ 303

IMPORTANT The following error was detected by w3af and couldn't be resolved: The remote web server is not answering our HTTP requests, multiple errors have been found while trying to GET a response from the server.

In most cases this means that the configured target is incorrect, the port is closed, there is a firewall blocking our packets or there is no HTTP daemon listening on that port.

Please verify your target configuration and try again.

Scan finished in 0 seconds.

[marrowsmith]

Let me know what you need next?

[marrowsmith]

Here is the edited file for reference:

    except Exception, ex:
        msg = ('Exception while inserting request/response to the'
               ' database: %s\nThe request/response that generated'
               ' the error is: %s %s %s' %
               (ex, resp.get_id(), request.get_uri(), resp.get_code()))
        om.out.error(msg)

raise Exception(msg)

        raise

[andresriancho]

Strange... I wasn't expecting this, will try to reproduce myself

[ghost]

no change got this error after editing

[ghost]

if somebody fix this please comment

[polyakoff]

I have the same problem. Need to mention that originally had version 1.2 of w3af on Kali linux and everything worked fine. UPD: Copied previously working ver 1.5 from my old laptop to the new one and it got the same error. But on the old laptop ver 1.5 worked like a charm. I have Kali linux on both machines. Hope this will help somehow.

[cryptable]

I'm running w3af on Kali Linux and have the same problem with 1.6. I have put some extra prints in the code, a backtrace at the exception (core/data/url/handlers/cache_backend/db.py) and the printout of 'data' variable before the msgpack.dump (core/data/db/history.py). Look at the header-key in the data dictionary, you'll see it starts Header(...) of the header object. I think msgpack has no way to handle objects like Header. Shouldn't the Header be a plain dictionary? Or there is a bug in our msgpack library.

Starting w3af, running on: Python version: 2.7.3 (default, Jan 2 2013, 13:56:14) [GCC 4.7.2] GTK version: 2.24.10 PyGTK version: 2.24.0 w3af version: w3af - Web Application Attack and Audit Framework Version: 1.6 Revision: 56914ed79c - 13 Oct 2013 16:12 Author: Andres Riancho and the w3af team. Enabling user_dir's dependency finger_bing Enabling user_dir's dependency finger_google Enabling user_dir's dependency finger_pks Enabling user_dir's dependency finger_bing Enabling user_dir's dependency finger_google Enabling user_dir's dependency finger_pks Called w3afCore.start() Enabling _dns_cache() Called buildopeners DNS response from DNS server for domain: XXXXXXXXXX ({'cookies': True, 'cache': False, 'uri': u'http://XXXXXXXXXXXXXX/', 'headers': Headers({u'Accept-encoding': u'gzip', u'User-agent': u'w3af.org', 'Host': '62.209.222.11', 'Accept': '/_'}), 'data': None, 'method': 'GET'}, {'body':... , 'code': 200, 'uri': u'http://XXXXXXXXXXXX/', 'headers': Headers({u'content-language': u'en-US', u'transfer-encoding': u'chunked', u'set-cookie': u'JSESSIONID=a8b00cd3-06c1-46f0-81d6-bea9ec74fe05; Path=/; HttpOnly', u'server': u'Apache/2.2.15 (CentOS)', u'connection': u'close', u'date': u'Sun, 03 Nov 2013 13:12:47 GMT', u'content-type': u'text/html;charset=UTF-8'}), 'time': 0.2, 'msg': 'OK', 'id': 1}) begin >>> traceback <<<< Traceback (most recent call last): File "/usr/share/w3af/core/data/url/handlers/cache_backend/db.py", line 82, in store_in_cache hi.save() File "/usr/share/w3af/core/data/db/history.py", line 321, in save msgpack.dump(data, req_res) File "_msgpack.pyx", line 164, in msgpack._msgpack.pack (msgpack/_msgpack.c:2228) Exception while inserting request/response to the database: Expected dict, got Headers The request/response that generated the error is: 1 http://XXXXXXXXXXX/ 200 File "_msgpack.pyx", line 153, in msgpack._msgpack.Packer.pack (msgpack/_msgpack.c:2020) File "_msgpack.pyx", line 142, in msgpack._msgpack.Packer._pack (msgpack/_msgpack.c:1862) File "_msgpack.pyx", line 136, in msgpack._msgpack.Packer._pack (msgpack/_msgpack.c:1761) File "_msgpack.pyx", line 130, in msgpack._msgpack.Packer._pack (msgpack/_msgpack.c:1611) TypeError: Expected dict, got Headers end >>> traceback <<<<

IMPORTANT The following error was detected by w3af and couldn't be resolved: The remote web server is not answering our HTTP requests, multiple errors have been found while trying to GET a response from the server.

In most cases this means that the configured target is incorrect, the port is closed, there is a firewall blocking our packets or there is no HTTP daemon listening on that port.

Please verify your target configuration and try again.

Scan finished in 0 seconds.

[cryptable]

Hi,

I think I found the problem: In core/data/url/HTTPRequest.py (line):

def to_dict(self):
    serializable_dict = {}
    sdict = serializable_dict

    sdict['method'], sdict['uri'] = self.get_method(), self.get_uri().url_string
    # sdict['headers'], sdict['data'] = self.get_headers(), self.get_data() -
    sdict['headers'], sdict['data'] = dict(self.get_headers()), self.get_data()
    sdict['cookies'] = self.cookies
    sdict['cache'] = self.get_from_cache

    return serializable_dict

Idem in core/data/url/HTTPResponse.py (line):

[andresriancho]

@ddt-tdd could you please confirm that by adding a dict() around self.get_headers() you don't get the error anymore? Which testing did you perform with that change?

[cryptable]

I don't get any errors anymore on that place, because I casted the OrderedDict to a normal dictionary. This removes the Header(...) stuff from the data variable. Now I have problems on another place. In the GUI mode the number req/s goes slowly to zero, but the CPU increases to 94% (using top). This has something to do with the google searches, but I'm investigating this issue.

[Nelis1982]

@ddt-tdd I can conform the option to include the dict() around the get_headers (in both HTTPRequest.py and HTTPResponse.py) solved did solve the DNS issue. Unfortunately I must also confirm the reqeuest/sec dropping to 0 during a scan. However it did not produced a high CPU. (~25%) / Running w3af 1.6 in Kali on VMware. During scanning, the GUI was also available and responded well. (which is to be expected with 0 request/sec)

In the terminal (from which I started w3af_gui) I noticed lots of GET requests pointing to the URL's discovered by the google search. Don't know if this is an error or if this works as designed. Have no idea why a scan on a defined target should lead to so many GET requests on other hosts.

Final notice is about google trowing the verification website at me requesting me to type some bmp letters in a textbox confirming I'm a human ;-). Happened to me after every scan I performed.

[andresriancho]

During the last days I've been working on having a continuous integration build of w3af. This is still not working, and won't work for at least 1 more month (I'm travelling and can't work on this too much).

What I want to do is:

• Have a working build (all tests pass)
• Write a test that reproduces the "Exception while inserting request/response to the database: Expected dict, got Headers" issue.
• Then apply your dict() change to make sure no regressions are added and that the test now passes.

You guys are more than welcome to help out with step number 2 (write a test that reproduces this). It should be a unittest like many of the ones we have in w3af (search for files named test_).

[cryptable]

Thanks,

I update the w3af in my Kali Linux and see if most problems are solved.

Greetings, David

2013/11/12 andresriancho notifications@github.com

During the last days I've been working on having a continuous integration build of w3af https://circleci.com/gh/andresriancho/w3af/100. This is still not working, and won't work for at least 1 more month (I'm travelling and can't work on this too much).

What I want to do is:

• Have a working build (all tests pass)
• Write a test that reproduces the "Exception while inserting request/response to the database: Expected dict, got Headers" issue.
• Then apply your dict() change to make sure no regressions are added and that the test now passes.

You guys are more than welcome to help out with step number 2 (write a test that reproduces this). It should be a unittest like many of the ones we have in w3af (search for files named test_).

— Reply to this email directly or view it on GitHubhttps://github.com/andresriancho/w3af/issues/665#issuecomment-28340127 .

[andresriancho]

Now that we have CI (see: http://w3af.org/how-w3af-uses-continuous-integration-to-improve) I'll work on this issue.

[andresriancho]

Seems that msgpack can handle Headers objects well (at least in this case):

>>> import msgpack
>>> from w3af.core.data.dc.headers import Headers
>>> x = ({'cookies': True, 'cache': False, 'uri': u'http://XXXXXXXXXXXXXX/', 
'headers': Headers({u'Accept-encoding': u'gzip', u'User-agent': u'w3af.org',
'Host': '62.209.222.11', 'Accept': '/'}.items()), 'data': None, 'method': 'GET'})
>>> msgpack.dumps(x)
'\x86\xa7...3GET'
>>> msgpack.loads(msgpack.dumps(x))
{'cookies': True, 'cache': False, 'uri': 'http://XXXXXXXXXXXXXX/', 'headers':
{'Host': '62.209.222.11', 'Accept-encoding': 'gzip', 'Accept': '/',
'User-agent': 'w3af.org'}, 'data': None, 'method': 'GET'}

[andresriancho]

Well, the build failed in the first try but passed later: https://circleci.com/gh/andresriancho/w3af/280

Everything looks good for now, hopefully this fixes the issue.

Please note that it's only fixed in the feature/module branch, which will be merged with develop shortly-

[ghost]

Works like a charm thanxx @ddt-tdd @andresriancho

[heayin]

hi andresriancho, you have said how to solve this problem here: https://circleci.com/gh/andresriancho/w3af/280 But when I jump to that address, I am not permitted to see how to solve this problem. I would appreciate it if you can tell me how I can have permission.

[andresriancho]

Translating that comment into something you can use:

cd w3af
git checkout feature/module
./w3af_console

[heayin]

It works fine after excecuting the cmd you have given , thanks a lot!

------------------ 原始邮件 ------------------ 发件人: "andresriancho";notifications@github.com; 发送时间: 2013年12月26日(星期四) 晚上8:05 收件人: "andresriancho/w3af"w3af@noreply.github.com; 抄送: "易 "21562936@qq.com; 主题: Re: [w3af] w3af 1.6 does not run correclty on Kali (#665)

Translating that comment into something you can use: cd w3af git checkout feature/module ./w3af_console
— Reply to this email directly or view it on GitHub.

[heayin]

hi andresriancho:

  I scan the dvwa website on metasploitable2. when i scan after configuring webspider(set the onlyforward option), this option does not work.I configure the target which is http://10.10.10.130/dvwa/(you can see it in the picture named config1.png), But the result has twiki(you can see int in the picture named reslutl1.png) . When I try to stop the scan ,it fails too.
 I tried to upload the config1.png and reslult1.png to https://github.com/andresriancho/w3af/. But it always failed. I have to send this email to show the question clearly.

------------------ 原始邮件 ------------------ 发件人: "andresriancho";notifications@github.com; 发送时间: 2013年12月26日(星期四) 晚上8:05 收件人: "andresriancho/w3af"w3af@noreply.github.com; 抄送: "易 "21562936@qq.com; 主题: Re: [w3af] w3af 1.6 does not run correclty on Kali (#665)

Translating that comment into something you can use: cd w3af git checkout feature/module ./w3af_console
— Reply to this email directly or view it on GitHub.

[heayin]

I use w3af in kali. the version of w3af is 1.6

------------------ 原始邮件 ------------------ 发件人: "易 ";21562936@qq.com; 发送时间: 2013年12月28日(星期六) 下午2:15 收件人: "andresriancho/w3af"reply@reply.github.com;

主题: 回复： [w3af] w3af 1.6 does not run correclty on Kali (#665)

hi andresriancho:

  I scan the dvwa website on metasploitable2. when i scan after configuring webspider(set the onlyforward option), this option does not work.I configure the target which is http://10.10.10.130/dvwa/(you can see it in the picture named config1.png), But the result has twiki(you can see int in the picture named reslutl1.png) . When I try to stop the scan ,it fails too.
 I tried to upload the config1.png and reslult1.png to https://github.com/andresriancho/w3af/. But it always failed. I have to send this email to show the question clearly.

------------------ 原始邮件 ------------------ 发件人: "andresriancho";notifications@github.com; 发送时间: 2013年12月26日(星期四) 晚上8:05 收件人: "andresriancho/w3af"w3af@noreply.github.com; 抄送: "易 "21562936@qq.com; 主题: Re: [w3af] w3af 1.6 does not run correclty on Kali (#665)

Translating that comment into something you can use: cd w3af git checkout feature/module ./w3af_console
— Reply to this email directly or view it on GitHub.

[andresriancho]

@heayin please create a different issue so I can handle your problem, it has nothing to do with Kali.-

[heayin]

I have create a new issue. Its address is as follows. https://github.com/andresriancho/w3af/issues/998 On the other hand, I have reported another issue. Its address is as follows. https://github.com/andresriancho/w3af/issues/996 I am a freshman. I wonder whether I should assign these issues to somebody or not.

------------------ 原始邮件 ------------------ 发件人: "andresriancho";notifications@github.com; 发送时间: 2013年12月30日(星期一) 上午8:30 收件人: "andresriancho/w3af"w3af@noreply.github.com; 抄送: "易 "21562936@qq.com; 主题: Re: [w3af] w3af 1.6 does not run correclty on Kali (#665)

@heayin please create a different issue so I can handle your problem, it has nothing to do with Kali.-

— Reply to this email directly or view it on GitHub.

[andresriancho]

Thanks for creating the different issues. No need to assign them to anyone.