Gvmd/gvm-cli will freak out on Nikto.pl results

[asmaack]

Not sure if this issue belongs here or with gvm-tools.

After a fresh install of Openvas/Gvm 11 on Ubuntu Server 18.04, I did a scan of some servers using scan config "Full and fast".

The scan will finish normally and the results are all fine and dandy to be viewed within gsa/gsad. However, when downloading the results with gvm-cli (gvm-tools) in XML format, everything will just crash in my face.

$ gvm-cli socket --raw --xml "<get_reports details=\"1\" report_id=\"2b62f841-1c63-4ffa-b742-9e46117ec816\" format_id=\"a994b278-1f62-11e1-96ac-406186ea4fc5\"/>"
('Cannot parse XML response. Response data read b\'t>Log</original_threat><original_severity>0</original_severity></result><result id="1290979b-a9eb-4b05-a9db-c53817c05cb1"><name>HTTP Server type and version</name><owner><name>admin</name></owner><modification_time>2020-01-29T10:32:55Z</modification_time><comment></comment><creation_time>2020-01-29T10:32:55Z</creation_time><host>192.168.1.1<asset asset_id=""/><hostname>_gateway</hostname></host><port>443/tcp</port><nvt oid="1.3.6.1.4.1.25623.1.0.10107"><type>nvt</type><name>HTTP Server type and version</name><family>Web Servers</family><cvss_base>0.0</cvss_base><tags>cvss_base_vector=AV:N/AC:L/Au:N/C:N/I:N/A:N|summary=This detects the HTTP Server&apos;s type and version.|insight=|affected=|impact=|solution=- Configure your server to use an alternate name like\\n  &apos;Wintendo httpD w/Dotmatrix display&apos;\\n\\n  - Be sure to remove common logos like apache_pb.gif.\\n\\n  - With Apache, you can set the directive &apos;ServerTokens Prod&apos; to limit\\n  the information emanating from the server in its response headers.|vuldetect=|solution_type=</tags></nvt>
. . . a lot of more stuff here . . .
+ The anti-clickjacking X-Frame-Options header is not present.\\n+ The X-XSS-Protection header is not defined. This header can hint to the user agent to protect against some forms of XSS\\n+ The site uses SSL and the Strict-Transport-Security HTTP header is not defined.\\n+ The site uses SSL and Expect-CT header is not present.\\n+ The X-Content-Type-Options header is not set. This could allow the user agent to render the content of the site in a different fashion to the MIME type\\n+ Root page / redirects to: https://192.168.1.129:6547/status;4a3a71d45d88607a=u8bflz8b20w1hj04sgqs7zaq\\n+ No CGI Directories found (use &apos;-C all&apos; to force check all possible dirs)\\n+ Hostname &apos;192.168.1.129&apos; does not match certificate&apos;s names: optiplex.bithex.com, optiplex, \\xc3\\x83&#x80;\\xc3\\x82\\xc2\\xa8&#x1;\\n, 192.168.1.10\\n+ 7915 requests: 0 error(s) and 6 item(s) reported on remote host\\n+ End Time:           2020-01-29 10:45:55 (GMT0) (196 seconds)\\n---------------------------------------------------------------------------\\n+ 1 host(s) tested\\n\\n</description><original_threat>Log</original_threat><original_severity>0</original_severity></result><result \'', XMLSyntaxError('xmlParseCharRef: invalid xmlChar value 1, line 834, column 56',))

The gvmd.log will output:

md   main:WARNING:2020-01-31 14h45.43 UTC:55875: read_from_client_unix: failed to read from client: Connection reset by peer

I noticed that the output ends in some data from Nikto.pl, so I tried disabling "Nikto (NASL wrapper)" within NVT family "Web application abuses" in the scan config, did the scan all over again and, lo and behold, the scan will finish and the results are fully downloadable with gvm-cli.

GVM versions

$ gvmd --version Greenbone Vulnerability Manager 9.0.1~git-3e3d2350-gvmd-9.0 GIT revision 3e3d2350-gvmd-9.0 Manager DB revision 221

$ openvas --version OpenVAS 7.0.0 gvm-libs 11.0.1

$ gvm-cli --version gvm-cli 2.1.0.dev1 (API version 1.2.0)

gvm-libs: $ git log commit 3ccb9244554afad949876b2f81a7901c32866a74 (HEAD -> gvm-libs-11.0, origin/gvm-libs-11.0)

Environment

Operating system: $ uname -a Linux xxx 5.3.0-28-generic #30~18.04.1-Ubuntu SMP Fri Jan 17 06:14:09 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 18.04.3 LTS Release: 18.04 Codename: bionic

Installation method / source: source installation

[Heartless68]

This bug is also present on distro Debian buster for source build. Mainly report format issue is related to this specific XML "a994b278-1f62-11e1-96ac-406186ea4fc5"

gvmd.log output is the exact same

Though in some experiences I found as well that the when using the gvm-tools CLI that during the report get command the report can be generated but it is mostly when the report isn't lean and contains the detail output.

In my experience/case example:

gvm-cli --gmp-username user --gmp-password password socket --socketpath=$HOME/gvm/var/run/gvmd.sock --xml='<get_reports report_id="GUID" format_id="a994b278-1f62-11e1-96ac-406186ea4fc5" details="1" notes_details="1" result_tags="1" ignore_pagination="1"/>'

It would error with the exact parse issue for the report

Then when doing this:

gvm-cli --gmp-username user --gmp-password password socket --socketpath=$HOME/gvm/var/run/gvmd.sock --xml='<get_reports report_id="GUID" format_id="a994b278-1f62-11e1-96ac-406186ea4fc5" notes_details="1" result_tags="1" ignore_pagination="1"/>'

Only the details="1" is not in the report generator from CLI then the parse error doesn't occur.

I don't want to open a duplicate of this bug, hence why I'm rather also commenting on your post.

I did see this bug was on the API for the Python-gvm as well: https://github.com/greenbone/python-gvm/issues/121

[drmendes]

https://community.greenbone.net/t/get-report-get-reports-yiedling-xml-syntax-and-parser-errors/4836

[Heartless68]

https://community.greenbone.net/t/get-report-get-reports-yiedling-xml-syntax-and-parser-errors/4836

Thank you for this link, and also the suggestion of the hammer solution same thing I was consider doing and attempting :) I'm assuming that it worked for you?

[drmendes]

Yes it does, despite being a provisional patch solution :| I believe the problem is on the gvmd's side while parsing the report.

[drmendes]

By the way, I had to change it to .replace('&', '') as the data is batched from the socket, resulting in the possibility of splitting & from the rest (#x or whatever) and not being replaced. This batching is the same reason why it is tough to use tools such html.escape.

[bjoernricks]

Should be fixed when running the command from #1014

Could somebody try the fix and give some feedback here?

[Heartless68]

Should be fixed when running the command from #1014

Could somebody try the fix and give some feedback here?

Nice, I'll check it out, will give update in few once build is done 👍

[Heartless68]

I'm weirdly getting an error with the osp_get_vts_version

gvmd is stating the following in the log:

lib   osp:WARNING:2020-03-18  14h00.04 utc:16637:  osp_get_vts_version: element VTS missing.
md manage:WARNING:2020-03-18  14h00.04 utc:16637:  manage_update_nvt_cache_osp:  failed to get scanner_version

However when I do run the following cli:

gvmd --verify-scanner 08b69003-5fc2-4037-a479-93b440211c73

It returns:

Scanner version: OpenVas 7.0.0.

I'm checking if I didn't make a mistake with the build at the moment

[Heartless68]

Okay I must have made a mistake somewhere, it is now updating again. I'm keeping an eye on the logs to ensure all info is loaded correctly.

Sorry about the roundabout here with this, was hoping for a quick build and test

[Heartless68]

So far can confirmed with two use cases where the get_report would've normally failed to parse correctly, it is now succeeding.

I'll run a few additional test, but it seems so far that it is working as expected.

[bjoernricks]

I'll run a few additional test, but it seems so far that it is working as expected.

sounds good already. we have implemented a fix for the current data in the postgres db (see https://github.com/greenbone/gvmd/pull/1014) and also some fixed in ospd(-openvas) e.g. https://github.com/greenbone/ospd/pull/226

[asmaack]

Somehow I cannot generate a report with Nikto.pl results in it. Sorry to say I had deleted the initial report (with Nikto.pl data) which caused the crash on my test server.

Just the same, I have updated a test system with gvm-libs, gvmd, done several scans, downloaded all with gvm-cli and all works very well it seems.

I will do some more playing around with the Nikto.pl wrapper later on.

[Heartless68]

@bjoernricks seems that the issue is still with a few results.

Here is one of the latest ones where the issue seems persistent:

• Input:

gvm-cli --gmp-username admin --gmp-password admin socket --socketpath=/home/admin/gvm/var/run/gvmd.sock --xml='<get_reports report_id="b3303953-0392-4ffd-9004-11d00cf33d3d" format_id="a994b278-1f62-11e1-96ac-406186ea4fc5" details="1" notes_details="1" result_tags="1" ignore_pagination="1"/>'

• Output:

<port>general/tcp</port><nvt oid="1.3.6.1.4.1.25623.1.0.108162"><type>nvt</type><name>Linux/UNIX SSH/LSC Authenticated Scan Info Consolidation</name><family>General</family><cvss_base>0.0</cvss_base><tags>cvss_base_vector=AV:N/AC:L/Au:N/C:N/I:N/A:N|summary=This script consolidates various technical information about\\n authenticated scans via SSH for Linux/UNIX targets.|insight=|affected=|impact=|solution=|vuldetect=|solution_type=</tags><refs><ref type="url" id="https://docs.greenbone.net/GSM-Manual/gos-4/en/vulnerabilitymanagement.html#requirements-on-target-systems-with-linux-unix"/><ref type="url" id="https://docs.greenbone.net/GSM-Manual/gos-5/en/scanning.html#requirements-on-target-systems-with-linux-unix"/><ref type="url" id="https://docs.greenbone.net/GSM-Manual/gos-6/en/scanning.html#requirements-on-target-systems-with-linux-unix"/></refs></nvt><scan_nvt_version></scan_nvt_version><threat>Log</threat><severity>0.0</severity><qod><value>80</value><type></type></qod><description>Description (Knowledge base entry) : Value/Content\\n--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\\nAlso use &apos;find&apos; command to search for Applications enabled within &apos;Options for Local Security Checks&apos; (ssh/lsc/enable_find) : yes\\nAmount of timeouts the &apos;find&apos; command has reached. (ssh/lsc/find_timeout) : None\\nClear received buffer before sending a command (ssh/force/clear_buffer) : FALSE\\nCommands are send via an pseudoterminal/pty (ssh/force/pty) : FALSE\\nDebugging enabled within &apos;Global variable settings&apos; (global_settings/ssh/debug) : FALSE\\nDescend directories on other filesystem enabled within &apos;Options for Local Security Checks&apos; (ssh/lsc/descend_ofs) : yes\\nDon&apos;t prepend &apos;/bin/sh -c&apos; to used commands (ssh/force/nosh) : FALSE\\nDon&apos;t prepend &apos;LANG=C; LC_ALL=C;&apos; to the &apos;/bin/sh -c&apos; commands (ssh/force/nolang_sh) : FALSE\\nFreeBSD patchlevel (ssh/login/freebsdpatchlevel) : Not applicable for target\\nFreeBSD release (ssh/login/freebsdrel) : Not applicable for target\\nLogin on a system with a restricted shell (ssh/restricted_shell) : FALSE\\nLogin on a system without common commands like &apos;cat&apos; or &apos;find&apos; (ssh/no_linux_shell) : FALSE\\nLogin via SSH failed (login/SSH/failed) : FALSE\\nLogin via SSH successful (login/SSH/success) : TRUE\\nMac OS X build (ssh/login/osx_build) : Not applicable for target\\nMac OS X release name (ssh/login/osx_name) : Not applicable for target\\nMac OS X version (ssh/login/osx_version) : Not applicable for target\\nMisconfigured CISCO device. No autocommand should be configured for the scanning user. (ssh/cisco/broken_autocommand) : FALSE\\nOpenBSD version (ssh/login/openbsdversion) : Not applicable for target\\nOperating System Key used (ssh/login/release) : None/Empty\\nPort used for authenciated scans (kb_ssh_transport()) : 22/tcp\\nReport vulnerabilities of inactive Linux Kernel(s) separately. (ssh/login/kernel_reporting_overwrite/enabled) : FALSE\\nResponse to &apos;uname -a&apos; command (ssh/login/uname) : FALSE\\nSend an extra command (ssh/send_extra_cmd) : FALSE\\nSolaris hardware type (ssh/login/solhardwaretype) : Not applicable for target\\nSolaris version (ssh/login/solosversion) : Not applicable for target\\nUser used for authenciated scans (kb_ssh_login()) : config\\nlocate: Command available (ssh/locate/available) : FALSE\\nlocate: Response to &apos;locate -S&apos; command (ssh/locate/broken) : \\n--------------- CONFIGURATION START ------------------\\nWelcome to the ProxySG appliance configuration wizard.\\n\\n\\nDirections:\\nFollow the on screen prompts to complete the initial configuration.\\nA restart may be required after completion of the initial configuration.\\n\\n\\nThis is a new Virtual Appliance\\nwhich must be assigned its own individual serial number.\\nSerial Number: /bin/sh -c&#x7;&#x7;&#x7;&#x7;&#x7;&#x7\'', XMLSyntaxError('xmlParseCharRef: invalid xmlChar value 7, line 17, column 31'))

(I had to truncate the output just so that the NVT and error spot is remaining)

[bjoernricks]

@Heartless68 Does this originate from a new scan?

[Heartless68]

@bjoernricks yes it was a brand new scan that got ran using the changes implemented.

[bjoernricks]

@jjnicola could you take a look again? The xml issue seems still not be fixed completely.

[jjnicola]

Hi @Heartless68,

I was testing this, and I was able to get it working with the master branch. The fixes were backported to the GVM-11 branches and it should be working as well. The test I performed was just to add raw data in the nasl script result message (as it was the message sent by the server). The problematic chars are those control chars like &#x7. OSPD is already handling this especial chars. Therefore, you should see in your report something like \x0007 instead.

Are you using the right new report_id. If you used the old report_id, the results can be still with non escaped chars and you will get the error. If the report is the right one (a new generated one), are the patches applied? it looks like you are not getting the escaped chars in your report.

If you want to try in your environment, what I did for the test is:

• modify a script you are sure that it will generate a result. I have scan config with a single NVT ssh_authentication.nasl. I replaced the "log_message" at the end of the script with the following lines (add control chars in raw format):

somehexa = raw_string(0x00, 0x04, 0x07, 0x01, 0x90, 0x00, "Bad Request", 0x00, 0x00); Data='It was possible to <xml> \n\n' + somehexa; security_message( port:port, data:Data );

• run the authenticated scan and check the results.

I was able to see in GSA results, creating the result as XML and downloading it (it generates the the report with format_id a994b278-1f62-11e1-96ac-406186ea4fc5), and also using gvm-cli to get the report as you wrote in your example. All was successful and I can see the escaped chars.

... <type/></qod><description>It was possible to &lt;xml&gt; \x0004\x0007\x0001&#xC2;\x0090 Bad Request </description><original_threat>Log</original_threat><original_severity>0</original_severity></result> ...

It would be nice if you can test this and give me some feedback. In case the issue persists, please send me the response sent by the server in raw format, if you have it and if possible.

[y0urself]

So since we didn't get any feedback, I will close it for now. If any related bugs occur please create a new issue.