ivre installed on kali does not show data in web gui

litebito commented 1 year ago

IVRE Version

Insert here the output of the command ivre version, displayed as code. For example:

IVRE - Network recon framework
Copyright 2011 - 2022 Pierre LALET <pierre@droids-corp.org>
Version 0.9.18-kali

Python 3.10.8 (main, Oct 24 2022, 10:07:16) [GCC 12.2.0]

Linux sec02-kali 5.19.0-kali2-amd64 #1 SMP PREEMPT_DYNAMIC Debian 5.19.11-1kali2 (2022-10-10) x86_64

Dependencies:
    Python module pymongo: 3.11.0
    Python module sqlalchemy: 1.4.31
    Python module psycopg2: 2.9.4 (dt dec pq3 ext lo64)
    Python module cryptography: 3.4.8
    Python module krbV: missing
    Python module pycurl: PycURL/7.45.1 libcurl/7.85.0 GnuTLS/3.7.8 zlib/1.2.11 brotli/1.0.9 zstd/1.5.2 libidn2/2.3.3 libpsl/0.21.0 (+libidn2/2.3.0) libssh2/1.10.0 nghttp2/1.50.0 librtmp/2.3
    Python module PIL: 9.2.0
    Python module MySQLdb: 1.4.6
    Python module dbus: 1.3.2
    Python module matplotlib: 3.5.2
    Python module bottle: 0.12.23
    Python module OpenSSL: 21.0.0
    Python module tinydb: 3.15.2

Summary

I'm trying to view data from ivre in the ivre-web-gui. I followed the steps to install ivre on kali (sudo apt install ivre) and the steps to initialise. Then I ran my first scan, and followed the steps to import the data into the db, and create the view Then I started the ivre webserver

Expected behavior

I expected the ivre installation on kali to automatically create/setup already a webserver When opening the browser to the webserver, I expected to see at least some of the scanned data.

Actual behavior

The installation did not create or setup a webserver, so I'm using the ivre httpd command to test the webgui The web gui is available, but remains empty, it shows the menu and filter options, but in the upper left corner it shows "NO RESULTS"

What am I missing to be able to view the data in the webgui?

How to reproduce

└─$ sudo ivre ipinfo --init                           
This will remove any passive information in your database. Process ? [y/N] y

└─$ ivre ipinfo --init 
This will remove any passive information in your database. Process ? [y/N] y

└─$ ivre scancli --init   
This will remove any scan result in your database. Process ? [y/N] y

└─$ ivre view --init   
This will remove any view in your database. Process ? [y/N] y

└─$ ivre flowcli --init
This will remove any flow result in your database. Process ? [y/N] y

└─$ sudo ivre runscansagentdb --init                                                                          
This will remove any agent and/or scan in your database and files. Process? [y/N] y

└─$ sudo ivre ipdata --download
Downloading https://ivre.rocks/data/geolite/GeoLite2-City.tar.gz to /usr/share/ivre/geoip/GeoLite2-City.tar.gz: done.
Downloading https://ivre.rocks/data/geolite/GeoLite2-City-CSV.zip to /usr/share/ivre/geoip/GeoLite2-City-CSV.zip: done.
Downloading https://ivre.rocks/data/geolite/GeoLite2-Country.tar.gz to /usr/share/ivre/geoip/GeoLite2-Country.tar.gz: done.
Downloading https://ivre.rocks/data/geolite/GeoLite2-Country-CSV.zip to /usr/share/ivre/geoip/GeoLite2-Country-CSV.zip: done.
Downloading https://ivre.rocks/data/geolite/GeoLite2-ASN.tar.gz to /usr/share/ivre/geoip/GeoLite2-ASN.tar.gz: done.
Downloading https://ivre.rocks/data/geolite/GeoLite2-ASN-CSV.zip to /usr/share/ivre/geoip/GeoLite2-ASN-CSV.zip: done.
Downloading https://ivre.rocks/data/geolite/GeoLite2-dumps.tar.gz to /usr/share/ivre/geoip/GeoLite2-dumps.tar.gz: done.
Downloading https://dev.maxmind.com/csv-files/codes/iso3166.csv to /usr/share/ivre/geoip/iso3166.csv: done.
Downloading http://thyme.apnic.net/current/data-raw-table to /usr/share/ivre/geoip/BGP.raw: done.
Unpacking: <function gunzip at 0x7fce845cbe20>(*['GeoLite2-City.tar.gz'], **{})
<function untar_all at 0x7fce845cbeb0>(*['GeoLite2-City.tar'], **{'cond': <function <lambda> at 0x7fce845d01f0>})
<function gunzip at 0x7fce845cbe20>(*['GeoLite2-dumps.tar.gz'], **{})
<function untar_all at 0x7fce845cbeb0>(*['GeoLite2-dumps.tar'], **{'cond': <function <lambda> at 0x7fce845d03a0>})
<function unzip_all at 0x7fce845cbd90>(*['GeoLite2-ASN-CSV.zip'], **{'cond': <function <lambda> at 0x7fce845d0160>})
<function unzip_all at 0x7fce845cbd90>(*['GeoLite2-Country-CSV.zip'], **{'cond': <function <lambda> at 0x7fce845d00d0>})
<function bgp_raw_to_csv at 0x7fce84571000>(*['BGP.raw', 'BGP.csv'], **{})
<function gunzip at 0x7fce845cbe20>(*['GeoLite2-Country.tar.gz'], **{})
<function untar_all at 0x7fce845cbeb0>(*['GeoLite2-Country.tar'], **{'cond': <function <lambda> at 0x7fce845d0280>})
<function unzip_all at 0x7fce845cbd90>(*['GeoLite2-City-CSV.zip'], **{'cond': <function <lambda> at 0x7fce845d0040>})
<function gunzip at 0x7fce845cbe20>(*['GeoLite2-ASN.tar.gz'], **{})
<function untar_all at 0x7fce845cbeb0>(*['GeoLite2-ASN.tar'], **{'cond': <function <lambda> at 0x7fce845d0310>})
done.

└─$ sudo ivre getwebdata       

└─$ sudo ivre ipdata --import-all 
INFO:ivre:Dumping 'GeoLite2-ASN.mmdb' to 'GeoLite2-ASN.dump-IPv4.csv'
INFO:ivre:Dumping 'GeoLite2-Country.mmdb' to 'GeoLite2-Country.dump-IPv4.csv'
INFO:ivre:Dumping 'GeoLite2-Country.mmdb' to 'GeoLite2-RegisteredCountry.dump-IPv4.csv'
INFO:ivre:Skipping 'GeoLite2-City.mmdb' since 'GeoLite2-City.dump-IPv4.csv' is newer

└─$ sudo ivre runscans --routable --network 10.0.2.0/255.255.255.0 --limit 255 --output=XMLFork --processes 10

└─$ ivre scan2db -c LANS -s MYLANSCAN -r scans/NET-10.0.2.0-255.255.255.0                                     
INFO:ivre:New OS not yet registered in WINDOWS_VERSION_TO_BUILD 'QTS'
INFO:ivre:New OS not yet registered in WINDOWS_VERSION_TO_BUILD 'QTS'
INFO:ivre:256 results imported.

└─$ ivre db2view nmap                                                    

└─$ ivre db2view nmap                                                    

└─$ ivre view --count
58

└─$ sudo ivre httpd --bind-address 0.0.0.0 --port 8081

This program runs a simple httpd server to provide an out-of-the-box
access to the web user interface.

This script should only be used for testing purposes. Production
deployments should use "real" web servers (IVRE has been successfully
tested with both Apache and Nginx).

Bottle v0.12.23 server starting up (using WSGIRefServer())...
Listening on http://0.0.0.0:8081/
Hit Ctrl-C to quit.

127.0.0.1 - - [29/Oct/2022 16:17:59] "GET /index.html HTTP/1.1" 304 0
127.0.0.1 - - [29/Oct/2022 16:18:00] "GET /cgi/config HTTP/1.1" 200 311
127.0.0.1 - - [29/Oct/2022 16:18:00] "GET /cgi/view/count?q=&callback=jQuery203014041984525951223_1667053080222&_=1667053080223 HTTP/1.1" 200 46
127.0.0.1 - - [29/Oct/2022 16:18:00] "GET /cgi/view?q=&callback=jQuery203014041984525951223_1667053080224&_=1667053080225 HTTP/1.1" 200 98

More

$ cat /etc/ivre/ivre.conf
DB="tinydb://%s/.ivre" % os.path.expanduser("~")

p-l- commented 1 year ago

Hi,

Thanks for your detailed report.

Kali defaults to TinyDB backend, which is a file-backed database. Which has several drawbacks, one of which is: you cannot use the same database with different users.

This is explicitly said in the configuration file (os.path.expanduser("~") will evaluate to the home directory of the user running it, which is usually different from a user to another).

This means that if you use ivre view --count to check how many results exist in your database, you should use ivre httpd --bind-address 0.0.0.0 --port 8081 (without sudo) to serve the same results.

A couple of remarks:

If you end up with a lot of results, you may want to switch to the default MongoDB backend.
You should not run the web server as root anyway! Instead, consider using a dedicated, low-privileged user (so that in case of a vulnerability, only that user is affected).

Don't hesitate to join the conversation on Gitter if you have further questions.

litebito commented 1 year ago

Hi

thanks for the feedback.... I did try to run the webserver as non root the first time, but it was also empty. I did all of the steps again (initializing) and now it indeed seems to work, thanks!

If I understand you correctly, you suggest not to use the Kali package, but to use the manual installation for IVRE also on Kali?

p-l- commented 1 year ago

No, you can use the kali package (unless you need new features, that have not yet been integrated to a release), but also install MongoDB and change IVRE configuration to use it (basically, removing /etc/ivre/ivre.conf would work for a local, unauthenticated database).

p-l- commented 1 year ago

Also, since your issue has been fixed, I'm closing this.

ivre / ivre