search

mushorg / glastopf

Web Application Honeypot
http://glastopf.org
555 stars 169 forks source link

events not added to database #126

Closed peikkko closed 10 years ago

peikkko commented 11 years ago

Hi,

I installed latest glastopf it is running fine but I am wondering why only some events are logged into the database (no matter if mysql or sqlite). On console log and within the logfiles I can see a lot more events. As I also do not get error messages, I do not have an idea, how to investigate this.

johnnykv commented 11 years ago

Only attack events are logged. From the example below only the very last line is stored in the database:

2013-03-14 09:17:59,004 (glastopf.glastopf) Initializing Glastopf using "/opt/myhoneypot" as work directory.
2013-03-14 09:17:59,014 (glastopf.glastopf) Connecting to main database with: sqlite:///db/glastopf.db
2013-03-14 09:17:59,113 (glastopf.modules.handlers.emulators.dork_list.dork_page_generator) Bootstrapping dork database.
2013-03-14 09:17:59,133 (requests.packages.urllib3.connectionpool) Starting new HTTPS connection (1): mnemosyne.honeycloud.net
2013-03-14 09:18:00,154 (requests.packages.urllib3.connectionpool) "POST /login HTTP/1.1" 200 30
2013-03-14 09:18:00,589 (requests.packages.urllib3.connectionpool) "GET /api/v1/aux/dorks?limit=1000 HTTP/1.1" 200 180459
2013-03-14 09:18:00,711 (glastopf.modules.handlers.emulators.dork_list.mnem_service) Successfully retrieved 1000 dorks from the mnemosyne service.
2013-03-14 09:18:02,752 (glastopf.glastopf) Generating initial dork pages - this can take a while.
2013-03-14 09:18:05,223 (glastopf.modules.reporting.auxiliary.log_hpfeeds) Connecting to feed broker.
2013-03-14 09:18:05,280 (glastopf.modules.reporting.auxiliary.log_hpfeeds) Connected to hpfeed broker.
2013-03-14 09:18:08,408 (glastopf.glastopf) Glastopf started and privileges dropped.
2013-03-14 09:18:49,185 (glastopf.glastopf) 172.16.177.131 requested GET / on 172.16.177.131
peikkko commented 11 years ago

this is what I got in the logs:

2013-09-12 13:57:49,424 (glastopf.glastopf) Initializing Glastopf 3.0.9-dev using "/opt/myhoneypot" as work directory. 2013-09-12 13:57:49,426 (glastopf.glastopf) Connecting to main database with: mysql://user:pass@localhost/glaspot 2013-09-12 13:57:49,491 (pyhpfeeds) connecting to hpfriends.honeycloud.net:20000 2013-09-12 13:57:49,775 (pyhpfeeds) info message name: hpfriends, rand: '\xcaXW\x8e' 2013-09-12 13:57:49,789 (glastopf.glastopf) Glastopf started and privileges dropped. 2013-09-12 14:16:52,962 (glastopf.glastopf) 66.249.78.115 requested GET /robots.txt on www.example.com 2013-09-12 14:16:52,989 (pyhpfeeds) Socket error: [Errno 32] Broken pipe 2013-09-12 14:16:52,989 (pyhpfeeds) Disconnected from broker (in publish). 2013-09-12 14:16:52,989 (pyhpfeeds) connecting to hpfriends.honeycloud.net:20000 2013-09-12 14:16:53,175 (glastopf.glastopf) 66.249.78.115 requested GET / on www.example.com 2013-09-12 14:16:53,267 (pyhpfeeds) info message name: hpfriends, rand: '\x93lY\xbb' 2013-09-12 14:16:57,558 (glastopf.glastopf) 66.249.78.115 requested GET /index.php?site= on www.example.com 2013-09-12 14:16:57,985 (glastopf.glastopf) 66.249.78.115 requested GET /home.php?panel= on www.example.com 2013-09-12 14:16:58,482 (glastopf.glastopf) 66.249.78.115 requested GET /gallery.php?pollname= on www.example.com 2013-09-12 14:16:59,104 (glastopf.glastopf) 66.249.78.115 requested GET /confixx on www.example.com 2013-09-12 14:16:59,865 (glastopf.glastopf) 66.249.78.115 requested GET /backup on www.example.com 2013-09-12 14:17:00,620 (glastopf.glastopf) 66.249.78.115 requested GET /adm-cfgedit.php on www.example.com 2013-09-12 14:17:01,368 (glastopf.glastopf) 66.249.78.115 requested GET /show.php?l= on www.example.com 2013-09-12 14:17:02,111 (glastopf.glastopf) 66.249.78.115 requested GET /down%2A.php?z= on www.example.com 2013-09-12 14:17:02,851 (glastopf.glastopf) 66.249.78.115 requested GET /show.php?go= on www.example.com

but only 1 event was logged in the database

| 5 | 2013-09-12 14:16:52 | 66.249.78.115:60682 | /robots.txt | GET /robots.txt HTTP/1.1 Accept: text/plain,text/html Accept-Encoding: gzip,deflate Connection: Keep-alive From: googlebot(at)googlebot.com Host: www.example.com User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | robots | NULL | +----+---------------------+---------------------+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+----------+

I would expect all of these attempts would be logged into database?

johnnykv commented 11 years ago

Uhrm, that's funny. Any chance that you could send me your sqlite data? Do you use glastopf from pypi or github?

peikkko commented 11 years ago

I installed it via github. I also do not understand why these lines are missing:

2013-03-14 09:17:59,113 (glastopf.modules.handlers.emulators.dork_list.dork_page_generator) Bootstrapping dork database. 2013-03-14 09:17:59,133 (requests.packages.urllib3.connectionpool) Starting new HTTPS connection (1): mnemosyne.honeycloud.net 2013-03-14 09:18:00,154 (requests.packages.urllib3.connectionpool) "POST /login HTTP/1.1" 200 30 2013-03-14 09:18:00,589 (requests.packages.urllib3.connectionpool) "GET /api/v1/aux/dorks?limit=1000 HTTP/1.1" 200 180459 2013-03-14 09:18:00,711 (glastopf.modules.handlers.emulators.dork_list.mnem_service) Successfully retrieved 1000 dorks from the mnemosyne service. 2013-03-14 09:18:02,752 (glastopf.glastopf) Generating initial dork pages - this can take a while. 2013-03-14 09:18:05,223 (glastopf.modules.reporting.auxiliary.log_hpfeeds) Connecting to feed broker.

this is my glastopf.cfg

[webserver] host = 0.0.0.0 port = 80 uid = nobody gid = nogroup proxy_enabled = False

Generic logging for general monitoring

[logging] consolelog_enabled = True filelog_enabled = True logfile = log/glastopf.log

[dork-db] enabled = True pattern = rfi token_pattern = /\w+

parameters for clustering of dorks (KMeans)

n_clusters = 10 max_iter = 50 n_init = 20

Extracts dorks from a online dorks service operated by The Honeynet Project

mnem_service = True

[hpfeed] enabled = true host = hpfriends.honeycloud.net port = 20000 secret = 3wis3l2u5l7r3cew

channels comma separated

chan_events = glastopf.events chan_files = glastopf.files [main-database]

If disabled a sqlite database will be created (db/glastopf.db)

to be used as dork storage.

enabled = True

mongodb or sqlalchemy connection string, ex:

mongodb://localhost:27017/glastopf

mongodb://james:bond@localhost:27017/glastopf

mysql://james:bond@somehost.com/glastopf

connection_string = sqlite:///db/glastopf.db connection_string = mysql://user:pass@localhost/glaspot

[surfcertids] enabled = False host = localhost port = 5432 user = password = database = idsserver

[syslog] enabled = true socket = /dev/log

[mail] enabled = False patterns = rfi,lfi user = pwd = mail_from = mail_to = smtp_host = smtp.gmail.com smtp_port = 587

[misc] banner = Apache/2.0.48

peikkko commented 11 years ago

I saw this already with 3.0.8 - this was installed with pip, that's why I installed 3.0.9-dev because I thought this would fix this

johnnykv commented 11 years ago

@glaslos @nixcon Have any of you guys noticed this bug? If so, could you take a look at it?

gutehall commented 11 years ago

@johnnykv Nope, haven't noticed it before. I'll check my database and get back.

peikkko commented 10 years ago

Hi,

did you find anything about this issue? I just did an update of glastopf but still the same...

Shortfinga commented 10 years ago

Hi, I have the same issue. My last log in the database was: 2013-11-18 04:01:31,981 (glastopf.glastopf) 199.58.86.XXX requested GET /phpwcms/include/inc_ext/spaw/dialogs/components/com_galleria/down%2A.php?path= on www..com The event one second later isn't in the database any more. Very strange! I restarted the hole server and now it works again. Version: Glastopf 3.0.8 weather pip or github, I can't say.

Thank you in advance Shortfinga

johnnykv commented 10 years ago

@Shortfinga Thanks for the bug report. Could you please provide the following information;

Shortfinga commented 10 years ago

The sqlalchemy-version is 0.7.8 There was about 300GB of free disk space I think the server inclusive glastopf was running 25 days

[code]

id time source request_url request_raw pattern filename
111014 2013-11-18 04:01:27 199.58.86.XXX:51590 /phpwcms/include/inc_ext/spaw/dialogs/components/com_galleria/default.php?pr= GET /phpwcms/include/inc_ext/spaw/dialogs/components/com_galleria/default.php?pr= HTTP/1.0 Accept: text/html,text/plain,text/xml,text/*,application/xml,application/xhtml+xml,application/rss+xml,application/atom+xml,application/rdf+xml Accept-Encoding: gzip Accept-Language: en Connection: close Host: www.guiacreativity.com User-Agent: Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+) unknown None
111015 2013-11-18 04:01:28 199.58.86.XXX:52351 /phpwcms/include/inc_ext/spaw/dialogs/components/com_galleria/default.php?seite= GET /phpwcms/include/inc_ext/spaw/dialogs/components/com_galleria/default.php?seite= HTTP/1.0 Accept: text/html,text/plain,text/xml,text/*,application/xml,application/xhtml+xml,application/rss+xml,application/atom+xml,application/rdf+xml Accept-Encoding: gzip Accept-Language: en Connection: close Host: www.guiacreativity.com User-Agent: Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+) unknown None
111016 2013-11-18 04:01:29 124.73.140.XXX:56899 /register.php GET /register.php HTTP/1.1 Content-Type: application/x-www-form-urlencoded Host: junta.drupal.cat Referer: http://junta.drupal.cat/register.php User-Agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.57 Safari/537.17 unknown None
111017 2013-11-18 04:01:30 199.58.86.XXX:53171 /phpwcms/include/inc_ext/spaw/dialogs/components/com_galleria/down%2A.php?incl= GET /phpwcms/include/inc_ext/spaw/dialogs/components/com_galleria/down%2A.php?incl= HTTP/1.0 Accept: text/html,text/plain,text/xml,text/*,application/xml,application/xhtml+xml,application/rss+xml,application/atom+xml,application/rdf+xml Accept-Encoding: gzip Accept-Language: en Connection: close Host: www.guiacreativity.com User-Agent: Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+) unknown None

[/code]

johnnykv commented 10 years ago

Thanks i will look into this issue again. Did you have hpfeeds enabled by chance?

On 24/11/2013, at 20.52, Shortfinga notifications@github.com wrote:

The sqlalchemy-version is 0.7.8 There was about 300GB of free disk space I think the server inclusive glastopf was running 25 days

[code]

id time source request_url request_raw pattern filename 111014 2013-11-18 04:01:27 199.58.86.XXX:51590 /phpwcms/include/inc_ext/spaw/dialogs/components/com_galleria/default.php?pr= GET /phpwcms/include/inc_ext/spaw/dialogs/components/comgalleria/default.php?pr= HTTP/1.0 Accept: text/html,text/plain,text/xml,text/,application/xml,application/xhtml+xml,application/rss+xml,application/atom+xml,application/rdf+xml Accept-Encoding: gzip Accept-Language: en Connection: close Host: www.guiacreativity.com User-Agent: Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+) unknown None 111015 2013-11-18 04:01:28 199.58.86.XXX:52351 /phpwcms/include/inc_ext/spaw/dialogs/components/com_galleria/default.php?seite= GET /phpwcms/include/inc_ext/spaw/dialogs/components/comgalleria/default.php?seite= HTTP/1.0 Accept: text/html,text/plain,text/xml,text/,application/xml,application/xhtml+xml,application/rss+xml,application/atom+xml,application/rdf+xml Accept-Encoding: gzip Accept-Language: en Connection: close Host: www.guiacreativity.com User-Agent: Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+) unknown None 111016 2013-11-18 04:01:29 124.73.140.XXX:56899 /register.php GET /register.php HTTP/1.1 Content-Type: application/x-www-form-urlencoded Host: junta.drupal.cat Referer: http://junta.drupal.cat/register.php User-Agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.57 Safari/537.17 unknown None 111017 2013-11-18 04:01:30 199.58.86.XXX:53171 /phpwcms/include/inc_ext/spaw/dialogs/components/com_galleria/down%2A.php?incl= GET /phpwcms/include/inc_ext/spaw/dialogs/components/com_galleria/down%2A.php?incl= HTTP/1.0 Accept: text/html,text/plain,text/xml,text/*,application/xml,application/xhtml+xml,application/rss+xml,application/atom+xml,application/rdf+xml Accept-Encoding: gzip Accept-Language: en Connection: close Host: www.guiacreativity.com User-Agent: Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+) unknown

Shortfinga commented 10 years ago

Yep I have and had hpfeed enabled. I feed the default feed. (btw: could you provide me access to the hpfeed of glastopf?)

Shortfinga commented 10 years ago

Hi, same error again. Last entries in DB orderd by time (desc):

id time source request_url request_raw pattern filename
115996 2013-11-25 16:40:04 66.249.73.XXX:63030 /inc/cmses/blank.php?panel= GET /inc/cmses/blank.php?panel= HTTP/1.1
Accept: */*
Accept-Encoding: gzip,deflate
Connection: Keep-alive
From: googlebot(at)googlebot.com
Host: junta.drupal.cat
User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)		 unknown None
115995 2013-11-25 16:39:54 66.249.73.XXX:49933 /cgi-bin/sub%2A.php?lang= GET /cgi-bin/sub%2A.php?lang= HTTP/1.1
Accept: */*
Accept-Encoding: gzip,deflate
Connection: Keep-alive
From: googlebot(at)googlebot.com
Host: www.guiacreativity.com
User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)		 unknown None
115994 2013-11-25 16:39:24 66.249.73.XXX:41609 /%2A/_vti_pvt/file.php?body= GET /%2A/_vti_pvt/file.php?body= HTTP/1.1
Accept: */*
Accept-Encoding: gzip,deflate
Connection: Keep-alive
From: googlebot(at)googlebot.com
Host: www.guiacreativity.com
User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)		 unknown None
115993 2013-11-25 16:39:08 66.249.73.XXX:39037 /gnu3/mail GET /gnu3/mail HTTP/1.1
Accept: */*
Accept-Encoding: gzip,deflate
Connection: Keep-alive
From: googlebot(at)googlebot.com
Host: www.guiacreativity.com
User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)		 unknown None

Here is the log: 2013-11-25 16:39:08,335 (glastopf.glastopf) 66.249.73.XXX requested GET /gnu3/mail on www.guiacreativity.com 2013-11-25 16:39:24,420 (glastopf.glastopf) 66.249.73.XXX requested GET /%2A/_vti_pvt/file.php?body= on www.guiacreativity.com 2013-11-25 16:39:54,996 (glastopf.glastopf) 66.249.73.XXX requested GET /cgi-bin/sub%2A.php?lang= on www.guiacreativity.com 2013-11-25 16:40:04,016 (glastopf.glastopf) 66.249.73.XXX requested GET /inc/cmses/blank.php?panel= on junta.drupal.cat This is not in the DB anymore: 2013-11-25 16:40:08,267 (glastopf.glastopf) 78.172.96.XXX requested PUT /caspian.html on www.guiacreativity.com

Quite interesting that the entry which does not appear in the DB is a PUT Request. But I checked the log of the last time, there it was not a PUT-request:

2013-11-18 04:01:29,070 (glastopf.glastopf) 124.73.140.XXX requested GET /register.php on junta.drupal.cat 2013-11-18 04:01:30,383 (glastopf.glastopf) 199.58.86.XXX requested GET /phpwcms/include/inc_ext/spaw/dialogs/components/com_galleria/down%2A.php?incl= on www.guiacreativity.com This is not in the DB anymore: 2013-11-18 04:01:30,760 (glastopf.glastopf) 124.73.140.XXX requested POST /register.php on junta.drupal.cat 2013-11-18 04:01:31,981 (glastopf.glastopf) 199.58.86.XXX requested GET /phpwcms/include/inc_ext/spaw/dialogs/components/com_galleria/down%2A.php?path= on www.guiacreativity.com 2013-11-18 04:01:32,452 (glastopf.glastopf) 124.73.140.XXX requested POST /register.php on junta.drupal.cat 2013-11-18 04:01:33,642 (glastopf.glastopf) 199.58.86.XXX requested GET /phpwcms/include/inc_ext/spaw/dialogs/components/com_galleria/down%2A.php?phpbb_root_path= on www.guiacreativity.com

Disc-Space is there..., mysql-Server still responses to queries, even to insert queries.

johnnykv commented 10 years ago

I am pretty sure that you are seeing this because the thread responsible for reporting goes down, can you please do 

grep -i "Error inserting attack event into main database" glastopf.log

and tell if you got any hits on that message?

Please try running the development version of Glastopf, i just added some error handling for the logging part which potentially fixes the issue.

johnnykv commented 10 years ago

@Shortfinga In regards to getting access to hpfeeds data, you need to get in contact with @glaslos.

johnnykv commented 10 years ago

@Shortfinga did the last commit fix your issue?

Shortfinga commented 10 years ago

Sorry, but I could not try it yet. I tried to re install glastopf, but it failed. I opened a issue at the libinjection-page but have no response yet.