Error when main database is set to false

[nsmfoo]

As described in the config file, if you set main database, enable to "False", it should default to use the sqlite db, instead it borks.

If set to "True" it stores the data in the database

--- Error message

2015-06-18 23:31:38,189 (glastopf.glastopf) Main datbase has been disabled, dorks will be stored in: sqlite://db/glastopf.db

Traceback (most recent call last):
  File "/usr/local/bin/glastopf-runner", line 5, in <module>
    pkg_resources.run_script('Glastopf==3.1.3.dev0', 'glastopf-runner')
  File "/usr/local/lib/python2.7/dist-packages/distribute-0.6.35-py2.7.egg/pkg_resources.py", line 505, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/local/lib/python2.7/dist-packages/distribute-0.6.35-py2.7.egg/pkg_resources.py", line 1245, in run_script
    execfile(script_filename, namespace, namespace)
  File "/usr/local/lib/python2.7/dist-packages/Glastopf-3.1.3.dev0-py2.7.egg/EGG-INFO/scripts/glastopf-runner", line 91, in <module>
    honeypot = GlastopfHoneypot(work_dir=args.workdir)
  File "/usr/local/lib/python2.7/dist-packages/Glastopf-3.1.3.dev0-py2.7.egg/glastopf/glastopf.py", line 75, in __init__
    (self.maindb, self.dorkdb) = self.setup_main_database(conf_parser)
  File "/usr/local/lib/python2.7/dist-packages/Glastopf-3.1.3.dev0-py2.7.egg/glastopf/glastopf.py", line 167, in setup_main_database
    sqla_engine = create_engine("sqlite://db/glastopf.db")
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/__init__.py", line 332, in create_engine
    return strategy.create(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 72, in create
    (cargs, cparams) = dialect.create_connect_args(u)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/dialects/sqlite/pysqlite.py", line 317, in create_connect_args
    " sqlite:////absolute/path/to/file.db" % (url,))
sqlalchemy.exc.ArgumentError: Invalid SQLite URL: sqlite://db/glastopf.db

Valid SQLite URL forms are: sqlite:///:memory: (or, sqlite://) sqlite:///relative/path/to/file.db sqlite:////absolute/path/to/file.db

---

[glaslos]

Seems like all you have to do is add an additional forward slash in the default path :)

[nsmfoo]

I will push a fix for it

[katkad]

there is already a pull request https://github.com/glastopf/glastopf/pull/247

does your fix work ? doesn't end with error ? I had to do additional changes to make it work

[nsmfoo]

I overlooked that one, sorry.

The fix worked as per my commit. Verified on a clean OS and Glastopf installation.

Setting the [main-database], enable = False. Gives you the following:

• Main datbase has been disabled, dorks will be stored in: sqlite:///db/glastopf.db (typo is by design, will push a commit for that one) ...
• (glastopf.modules.handlers.emulators.dork_list.database_sqla) Done with insert of 1 dorks into the database.

And the data ends up in the db/glastopf.db

[glaslos]

fixed with #248

[katkad]

well, I end up with

2015-07-02 11:46:35,263 (glastopf.glastopf) Initializing Glastopf 3.1.3-dev using "/opt/glastopf" as work directory. 2015-07-02 11:46:35,317 (glastopf.glastopf) Main database has been disabled, dorks will be stored in: sqlite:///db/glastopf.db 2015-07-02 11:46:35,340 (glastopf.glastopf) Generating initial dork pages - this can take a while. Traceback (most recent call last): File "/usr/local/bin/glastopf-runner", line 5, in pkg_resources.run_script('Glastopf==3.1.3-dev', 'glastopf-runner') File "/usr/local/lib/python2.7/site-packages/pkg_resources.py", line 492, in run_script self.require(requires)[0].run_script(script_name, ns) File "/usr/local/lib/python2.7/site-packages/pkg_resources.py", line 1350, in run_script execfile(script_filename, namespace, namespace) File "/usr/lib/python2.7/site-packages/Glastopf-3.1.3_dev-py2.7.egg/EGG-INFO/scripts/glastopf-runner", line 91, in honeypot = GlastopfHoneypot(work_dir=args.workdir) File "/usr/lib/python2.7/site-packages/Glastopf-3.1.3_dev-py2.7.egg/glastopf/glastopf.py", line 81, in init self.dork_generator.regular_generate_dork(0) File "/usr/lib/python2.7/site-packages/Glastopf-3.1.3_dev-py2.7.egg/glastopf/modules/handlers/emulators/dork_list/dork_page_generator.py", line 123, in regular_generate_dork self._rotate_pages() File "/usr/lib/python2.7/site-packages/Glastopf-3.1.3_dev-py2.7.egg/glastopf/modules/handlers/emulators/dork_list/dork_page_generator.py", line 134, in _rotate_pages new_pages = self.generate_dork_pages() File "/usr/lib/python2.7/site-packages/Glastopf-3.1.3_dev-py2.7.egg/glastopf/modules/handlers/emulators/dork_list/dork_page_generator.py", line 81, in generate_dork_pages inurl_list = self.database.select_data() File "/usr/lib/python2.7/site-packages/Glastopf-3.1.3_dev-py2.7.egg/glastopf/modules/handlers/emulators/dork_list/database_sqla.py", line 44, in select_data data = self.get_pattern_requests_sql(pattern=pattern) File "/usr/lib/python2.7/site-packages/Glastopf-3.1.3_dev-py2.7.egg/glastopf/modules/handlers/emulators/dork_list/database_sqla.py", line 56, in get_pattern_requests_sql results = self.engine.connect().execute(sql, x=pattern).fetchall() File "/usr/local/lib/python2.7/site-packages/SQLAlchemy-0.9.7-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py", line 729, in execute return meth(self, multiparams, params) File "/usr/local/lib/python2.7/site-packages/SQLAlchemy-0.9.7-py2.7-linux-x86_64.egg/sqlalchemy/sql/elements.py", line 321, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/usr/local/lib/python2.7/site-packages/SQLAlchemy-0.9.7-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement compiled_sql, distilled_params File "/usr/local/lib/python2.7/site-packages/SQLAlchemy-0.9.7-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py", line 958, in _execute_context context) File "/usr/local/lib/python2.7/site-packages/SQLAlchemy-0.9.7-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1160, in _handle_dbapi_exception exc_info File "/usr/local/lib/python2.7/site-packages/SQLAlchemy-0.9.7-py2.7-linux-x86_64.egg/sqlalchemy/util/compat.py", line 199, in raise_from_cause reraise(type(exception), exception, tb=exc_tb) File "/usr/local/lib/python2.7/site-packages/SQLAlchemy-0.9.7-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py", line 951, in _execute_context context) File "/usr/local/lib/python2.7/site-packages/SQLAlchemy-0.9.7-py2.7-linux-x86_64.egg/sqlalchemy/engine/default.py", line 436, in do_execute cursor.execute(statement, parameters) sqlalchemy.exc.OperationalError: (OperationalError) no such table: events u'SELECT request_url FROM events WHERE pattern = ?' ('rfi',)

[katkad]

hm, works on Ubuntu. I will look into that

[katkad]

yeah, it fails, only if database is set to false from the beginning

if DB is set to false after there is current_dir/glastopf.cfg , table events is already created, but remains empty, because nothing will be inserted into it

[nsmfoo]

Did not try that variant. But would that not require that you have a glastopf.cfg (and the rest of the file structure) to start with. For a new installation, it is created during the first run of glastopf-runner? But maybe it's best if you push your fix for my fix anyway =)