search

mushorg / glastopf

Web Application Honeypot
http://glastopf.org
550 stars 170 forks source link

ValueError: sample larger than population #172

Closed frankplummer closed 10 years ago

frankplummer commented 10 years ago

Hi all,

Trying to use glastopf with MySQL. Upon launching glastopf-runner from the working directory, I shortly get the following crash:

2014-04-22 17:24:02,373 (glastopf.glastopf) Glastopf started and privileges dropped.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/gevent/greenlet.py", line 327, in run
    result = self._run(*self.args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/Glastopf-3.1.3_dev-py2.7.egg/glastopf/modules/handlers/emulators/dork_list/dork_page_generator.py", line 121, in regular_generate_dork
    self._rotate_pages()
  File "/usr/local/lib/python2.7/dist-packages/Glastopf-3.1.3_dev-py2.7.egg/glastopf/modules/handlers/emulators/dork_list/dork_page_generator.py", line 132, in _rotate_pages
    new_pages = self.generate_dork_pages()
  File "/usr/local/lib/python2.7/dist-packages/Glastopf-3.1.3_dev-py2.7.egg/glastopf/modules/handlers/emulators/dork_list/dork_page_generator.py", line 83, in generate_dork_pages
    dork_seeds = random.sample(self.database.get_dork_list('inurl'), INURL_MIN_SIZE)
  File "/usr/lib/python2.7/random.py", line 320, in sample
    raise ValueError("sample larger than population")
ValueError: sample larger than population
<Greenlet at 0x2505730: <bound method DorkPageGenerator.regular_generate_dork of <glastopf.modules.handlers.emulators.dork_list.dork_page_generator.DorkPageGenerator object at 0x214c710>>(30)> failed with ValueError

Traceback (most recent call last):
  File "/usr/local/bin/glastopf-runner", line 5, in <module>
    pkg_resources.run_script('Glastopf==3.1.3-dev', 'glastopf-runner')
  File "/usr/local/lib/python2.7/dist-packages/distribute-0.6.35-py2.7.egg/pkg_resources.py", line 505, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/local/lib/python2.7/dist-packages/distribute-0.6.35-py2.7.egg/pkg_resources.py", line 1245, in run_script
    execfile(script_filename, namespace, namespace)
  File "/usr/local/lib/python2.7/dist-packages/Glastopf-3.1.3_dev-py2.7.egg/EGG-INFO/scripts/glastopf-runner", line 89, in <module>
    gevent.joinall(workers, raise_error=True)
  File "/usr/local/lib/python2.7/dist-packages/gevent/greenlet.py", line 404, in joinall
    raise obj.exception
ValueError: sample larger than population

My glastopf.cfg [main-database] is as follows:

[main-database]
#If disabled a sqlite database will be created (db/glastopf.db)
#to be used as dork storage.
enabled = True
#mongodb or sqlalchemy connection string, ex:
#mongodb://localhost:27017/glastopf
#mongodb://james:bond@localhost:27017/glastopf
connection_string = mysql://glastopf:********@10.0.254.22/honeypot
#connection_string = sqlite:///db/glastopf.db

I can confirm the MySQL credentials are valid - glastopf is creating tables and populating them on first run. Strangely, if I comment out that line and uncomment the default SQLite line - glastopf-runner will run fine and dandy! My setup requires me to use MySQL so I would rather stick to that if at all possible..

Pulled from the repo today, so it's fresh as a daisy with all latest commits.

Not sure what's going on..!

johnnykv commented 10 years ago

Hi, thanks for the bug report.

There seems to be a difference in how our ORM operates in regards to sqlite/mysql. I just submitted a fix to the repo - please pull again and try it out.

If you got time, i would appreciate if you could submit debug logging (glastopf-runner -v) before and after you pull the new code.

frankplummer commented 10 years ago

Hi,

Before:

root@Honey:~/honeypot# glastopf-runner -v
2014-04-22 19:28:19,893 (glastopf.glastopf) Initializing Glastopf 3.1.3-dev using "/root/honeypot" as work directory.
2014-04-22 19:28:19,895 (glastopf.glastopf) Connecting to main database with: mysql://glastopf:password@10.0.254.22/honeypot
2014-04-22 19:28:19,943 (glastopf.modules.handlers.emulators.dork_list.database_sqla) Creating SQLite database.
2014-04-22 19:28:19,970 (glastopf.modules.handlers.emulators.dork_list.database_sqla) Returned 970 dorks from the database (starts with: None)
2014-04-22 19:28:19,978 (glastopf.modules.handlers.emulators.dork_list.dork_page_generator) Bootstrapping dork database.
2014-04-22 19:28:19,979 (glastopf.modules.handlers.emulators.dork_list.dork_page_generator) Bootstraping dork database.
2014-04-22 19:28:19,999 (requests.packages.urllib3.connectionpool) Starting new HTTPS connection (1): mnemosyne.honeycloud.net
2014-04-22 19:28:20,956 (requests.packages.urllib3.connectionpool) "POST /login HTTP/1.1" 200 30
2014-04-22 19:28:22,411 (requests.packages.urllib3.connectionpool) "GET /api/v1/aux/dorks?limit=1000 HTTP/1.1" 200 159044
2014-04-22 19:28:22,723 (glastopf.modules.handlers.emulators.dork_list.mnem_service) Successfully retrieved 1000 dorks from the mnemosyne service.
2014-04-22 19:28:22,848 (glastopf.modules.handlers.emulators.dork_list.database_sqla) Starting insert of 2088 dorks into the database.
2014-04-22 19:28:39,494 (glastopf.modules.handlers.emulators.dork_list.database_sqla) Done with insert of 2088 dorks into the database.
2014-04-22 19:28:39,496 (glastopf.modules.handlers.emulators.dork_list.dork_page_generator) Finished bootstrapping dork database.
2014-04-22 19:28:39,712 (glastopf.glastopf) Glastopf started and privileges dropped.
2014-04-22 19:28:39,843 (glastopf.modules.handlers.emulators.dork_list.database_sqla) Returned 970 dorks from the database (starts with: None)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/gevent/greenlet.py", line 327, in run
    result = self._run(*self.args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/Glastopf-3.1.3_dev-py2.7.egg/glastopf/modules/handlers/emulators/dork_list/dork_page_generator.py", line 121, in regular_generate_dork
    self._rotate_pages()
  File "/usr/local/lib/python2.7/dist-packages/Glastopf-3.1.3_dev-py2.7.egg/glastopf/modules/handlers/emulators/dork_list/dork_page_generator.py", line 132, in _rotate_pages
    new_pages = self.generate_dork_pages()
  File "/usr/local/lib/python2.7/dist-packages/Glastopf-3.1.3_dev-py2.7.egg/glastopf/modules/handlers/emulators/dork_list/dork_page_generator.py", line 83, in generate_dork_pages
    dork_seeds = random.sample(self.database.get_dork_list('inurl'), INURL_MIN_SIZE)
  File "/usr/lib/python2.7/random.py", line 320, in sample
    raise ValueError("sample larger than population")
ValueError: sample larger than population
<Greenlet at 0x3cb2730: <bound method DorkPageGenerator.regular_generate_dork of <glastopf.modules.handlers.emulators.dork_list.dork_page_generator.DorkPageGenerator object at 0x38f6710>>(30)> failed with ValueError

Traceback (most recent call last):
  File "/usr/local/bin/glastopf-runner", line 5, in <module>
    pkg_resources.run_script('Glastopf==3.1.3-dev', 'glastopf-runner')
  File "/usr/local/lib/python2.7/dist-packages/distribute-0.6.35-py2.7.egg/pkg_resources.py", line 505, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/local/lib/python2.7/dist-packages/distribute-0.6.35-py2.7.egg/pkg_resources.py", line 1245, in run_script
    execfile(script_filename, namespace, namespace)
  File "/usr/local/lib/python2.7/dist-packages/Glastopf-3.1.3_dev-py2.7.egg/EGG-INFO/scripts/glastopf-runner", line 89, in <module>
    gevent.joinall(workers, raise_error=True)
  File "/usr/local/lib/python2.7/dist-packages/gevent/greenlet.py", line 404, in joinall
    raise obj.exception
ValueError: sample larger than population

After:

root@Honey:~/honeypot# glastopf-runner -v
2014-04-22 19:30:13,107 (glastopf.glastopf) Initializing Glastopf 3.1.3-dev using "/root/honeypot" as work directory.
2014-04-22 19:30:13,110 (glastopf.glastopf) Connecting to main database with: mysql://glastopf:password@10.0.254.22/honeypot
2014-04-22 19:30:13,160 (glastopf.modules.handlers.emulators.dork_list.database_sqla) Creating SQLite database.
2014-04-22 19:30:13,189 (glastopf.modules.handlers.emulators.dork_list.database_sqla) Returned 970 dorks from the database (starts with: None)
2014-04-22 19:30:13,302 (glastopf.glastopf) Glastopf started and privileges dropped.
2014-04-22 19:30:13,371 (glastopf.modules.handlers.emulators.dork_list.database_sqla) Returned 970 dorks from the database (starts with: None)
2014-04-22 19:30:13,379 (glastopf.modules.handlers.emulators.dork_list.database_sqla) Returned 189 dorks from the database (starts with: None)
2014-04-22 19:30:13,385 (glastopf.modules.handlers.emulators.dork_list.database_sqla) Returned 346 dorks from the database (starts with: None)

2014-04-22 19:31:36,904 (glastopf.glastopf) 31.3.236.10 requested GET / on Honey:80
2014-04-22 19:31:37,441 (glastopf.glastopf) 31.3.236.10 requested GET /style.css on Honey:80
2014-04-22 19:31:37,471 (glastopf.modules.handlers.emulators.dork_list.database_sqla) Starting insert of 1 dorks into the database.
2014-04-22 19:31:37,607 (glastopf.modules.handlers.emulators.dork_list.database_sqla) Done with insert of 1 dorks into the database.
2014-04-22 19:31:37,607 (glastopf.modules.handlers.emulators.dork_list.database_sqla) New dorks inserted: {'inurl': 0}
2014-04-22 19:31:37,827 (glastopf.glastopf) 31.3.236.10 requested GET /favicon.ico on Honey:80
2014-04-22 19:31:38,207 (glastopf.glastopf) 31.3.236.10 requested GET /favicon.ico on Honey:80

Looks like your change could have fixed it.. Killing/running glastopf-runner, I cannot reproduce the original issue with the new pull.

Nicely done and many thanks! Sterling support!

johnnykv commented 10 years ago

No problem, we always enjoy bug reports! :-)