search

rapid7 / metasploit-framework

Metasploit Framework
https://www.metasploit.com/
Other
33.8k stars 13.9k forks source link

Using db_import with msfdb webservice says "success" but did not import hosts #13932

Closed pbarry-r7 closed 1 year ago

pbarry-r7 commented 4 years ago

Using the most current version of Metasploit Framework in GitHub (specifically this hash), if I use msfdb to create a new db-via-webservice and then do a db_import via msfconsole of an Acunetix XML file that has one host in it, I'm told "success" but nothing was imported:

msf5 > db_status
[*] Connected to remote_data_service: (https://localhost:5443). Connection type: http. Connection name: local-https-data-service.
msf5 > db_import /vagrant/acunetix-2.xml
[*] Successfully imported /vagrant/acunetix-2.xml
msf5 > hosts

Hosts
=====

address  mac  name  os_name  os_flavor  os_sp  purpose  info  comments
-------  ---  ----  -------  ---------  -----  -------  ----  --------

If I do this BUT am using the "usual" direct-to-postgres db connection, the same db_import attempt works as expected (i.e. pulls in the one host and 3 services contained in my XML file):

msf5 > db_status
[*] Connected to msf_dev_db. Connection type: postgresql.
msf5 > db_import /vagrant/acunetix-2.xml
[*] Importing 'Acunetix' data
[*] Import: Parsing with 'Nokogiri v1.10.10'
[*] Importing host 21.1.24.14
[*] Importing service 21.1.24.14:80
[*] Importing service 21.1.24.14:21
[*] Importing service 21.1.24.14:22
[*] Successfully imported /vagrant/acunetix-2.xml
msf5 > hosts

Hosts
=====

address         mac  name             os_name                                 os_flavor  os_sp  purpose  info  comments
-------         ---  ----             -------                                 ---------  -----  -------  ----  --------    
21.1.24.14       server.info   Unknown                                                   device

I can provide the XML file I was working with on request.

Steps to reproduce

How'd you do it?

  1. msfdb delete to remove existing db
  2. msfdb init to create a new webservice db
  3. start msfconsole
  4. db_import <filename> for a supported file that contains at least one host
  5. observe success message
  6. type hosts and see no hosts

This section should also tell us any relevant information about the environment; for example, if an exploit that used to work is failing, tell us the victim operating system and service versions.

Were you following a specific guide/tutorial or reading documentation?

Nah, I was testing #13831 when I bumped into this.

Expected behavior

Data in the XML file should be imported to the msf webservice db.

Current behavior

Data is not imported from the XML into the msf webservice db.

You might also want to check the last ~1k lines of /opt/metasploit/apps/pro/engine/config/logs/framework.log or ~/.msf4/logs/framework.log for relevant stack traces

System stuff

Metasploit version

5.0.102-dev-d951c37 (GitHub current)

I installed Metasploit with:

OS

Ubuntu 16.04.6

jmartin-tech commented 4 years ago

Some interesting details in reproduction.

First attempt to import to a clean webservice & workspace some formats will be successful however exhibit delays when the data is loading.

msf5 > db_import m3_report.xml
[*] Successfully imported /home/vagrant/m3_report.xml
msf5 > hosts

Hosts
=====

address         mac  name  os_name  os_flavor  os_sp  purpose  info  comments
-------         ---  ----  -------  ---------  -----  -------  ----  --------
192.168.18.195

msf5 > hosts

Hosts
=====

address         mac                name            os_name  os_flavor  os_sp  purpose  info  comments
-------         ---                ----            -------  ---------  -----  -------  ----  --------
192.168.18.195  00:0C:29:6E:E8:7F  VAGRANT-2008R2

msf5 > hosts

Hosts
=====

address         mac                name            os_name  os_flavor  os_sp  purpose  info  comments
-------         ---                ----            -------  ---------  -----  -------  ----  --------
192.168.18.195  00:0C:29:6E:E8:7F  VAGRANT-2008R2

msf5 > hosts

Hosts
=====

address         mac                name            os_name  os_flavor  os_sp  purpose  info  comments
-------         ---                ----            -------  ---------  -----  -------  ----  --------
192.168.18.195  00:0C:29:6E:E8:7F  VAGRANT-2008R2

msf5 > hosts

Hosts
=====

address         mac                name            os_name  os_flavor  os_sp  purpose  info  comments
-------         ---                ----            -------  ---------  -----  -------  ----  --------
192.168.18.195  00:0C:29:6E:E8:7F  VAGRANT-2008R2

msf5 > hosts

Hosts
=====

address         mac                name            os_name                            os_flavor  os_sp  purpose  info  comments
-------         ---                ----            -------                            ---------  -----  -------  ----  --------
192.168.18.195  00:0c:29:6e:e8:7f  VAGRANT-2008R2  Windows 2008 R2, Standard Edition             SP1    server

msf5 > workspace -a second
[*] Added workspace: second
[*] Workspace: second
msf5 > db_import m3_report.xml
[*] Successfully imported /home/vagrant/m3_report.xml
msf5 > hosts

Hosts
=====

address         mac  name  os_name  os_flavor  os_sp  purpose  info  comments
-------         ---  ----  -------  ---------  -----  -------  ----  --------
192.168.18.195

msf5 > hosts

Hosts
=====

address         mac  name  os_name  os_flavor  os_sp  purpose  info  comments
-------         ---  ----  -------  ---------  -----  -------  ----  --------
192.168.18.195

msf5 > hosts

Hosts
=====

address         mac                name            os_name  os_flavor  os_sp  purpose  info  comments
-------         ---                ----            -------  ---------  -----  -------  ----  --------
192.168.18.195  00:0C:29:6E:E8:7F  VAGRANT-2008R2

msf5 > hosts

Hosts
=====

address         mac                name            os_name  os_flavor  os_sp  purpose  info  comments
-------         ---                ----            -------  ---------  -----  -------  ----  --------
192.168.18.195  00:0C:29:6E:E8:7F  VAGRANT-2008R2

msf5 > hosts

Hosts
=====

address         mac                name            os_name  os_flavor  os_sp  purpose  info  comments
-------         ---                ----            -------  ---------  -----  -------  ----  --------
192.168.18.195  00:0C:29:6E:E8:7F  VAGRANT-2008R2

msf5 >

However once an import has processed incorrectly such was when offered an invalid file, subsequent imports attempts for a valid file will result in no data.


$ msfconsole -q
msf5 > workspace -a third
[*] Added workspace: third
[*] Workspace: third
msf5 > db_import msfinstall
[*] Successfully imported /home/vagrant/msfinstall
msf5 > hosts

Hosts
=====

address  mac  name  os_name  os_flavor  os_sp  purpose  info  comments
-------  ---  ----  -------  ---------  -----  -------  ----  --------

msf5 > db_import m3_report.xml
[*] Successfully imported /home/vagrant/m3_report.xml
msf5 > hosts

Hosts
=====

address  mac  name  os_name  os_flavor  os_sp  purpose  info  comments
-------  ---  ----  -------  ---------  -----  -------  ----  --------

msf5 > hosts

Hosts
=====

address  mac  name  os_name  os_flavor  os_sp  purpose  info  comments
-------  ---  ----  -------  ---------  -----  -------  ----  --------

msf5 > hosts

Hosts
=====

address  mac  name  os_name  os_flavor  os_sp  purpose  info  comments
-------  ---  ----  -------  ---------  -----  -------  ----  --------

Valid sample import file attached.

m3_report.xml.zip

gwillcox-r7 commented 3 years ago

Encountered this error again whilst testing out stuff for https://github.com/rapid7/metasploit-framework/pull/14171. Pinging here as this is an issue we should really look at getting fixed given how often this could be encountered when reviewing database related PRs such as https://github.com/rapid7/metasploit-framework/pull/14171.

gwillcox-r7 commented 3 years ago

Note: Confirmed the temporary work around using db_disconnect to disconnect from the web services database and fall back to using the PostgreSQL database via a direct connection works. Ultimately though this issue will need to be solved at its root within the web services for a proper fix.

adfoster-r7 commented 1 year ago

Closing as this is the msfdb web service which isn't actively being developed

github-actions[bot] commented 1 year ago

Thanks for your contribution to Metasploit Framework! We've looked at this issue, and unfortunately we do not currently have the bandwidth to prioritize this issue.

We've labeled this as attic and closed it for now. If you believe this issue has been closed in error, or that it should be prioritized, please comment with additional information.

gwillcox-r7 commented 1 year ago

@adfoster-r7 Is the plan to ultimately remove support for the msfdb web service? Seems there are a number of bugs related to its usage at the moment, so I'm just curious our approach to that given the response here.

adfoster-r7 commented 1 year ago

@gwillcox-r7 It's not actively being removed - but it's now no longer enabled by default, and the user has to opt into the functionality now.

gwillcox-r7 commented 1 year ago

@gwillcox-r7 It's not actively being removed - but it's now no longer enabled by default, and the user has to opt into the functionality now.

Ah that's a shame, but makes sense.