Database is locked error when using FIFOSQLiteQueue

[mikehacksthings]

The fingerprint_targets process fails when using FIFOSQLiteQueue with: Exception: OperationalError: database is locked

[11:14:08][scan_engine][_build_targets] 39 fingerprints [11:14:08][scan_engine][scan] Number of procs: 10 [11:14:08][scan_engine][fingerprint_targets] 39 fingerprints remaining [11:14:18][scan_engine][fingerprint_targets] Caught exception: OperationalError [11:14:18][scan_engine][fingerprint_targets] Exception: OperationalError: database is locked [11:14:18][scan_engine][fingerprint_targets] 39 fingerprints remaining [11:14:28][scan_engine][fingerprint_targets] Caught exception: OperationalError [11:14:28][scan_engine][fingerprint_targets] Exception: OperationalError: database is locked [11:14:28][scan_engine][fingerprint_targets] 39 fingerprints remaining [11:14:38][scan_engine][fingerprint_targets] Caught exception: OperationalError [11:14:38][scan_engine][fingerprint_targets] Exception: OperationalError: database is locked [11:14:38][scan_engine][fingerprint_targets] 39 fingerprints remaining

Specific problem occurs on the fpq.get() call in fingerprint_targets()

[ztgrace]

On some systems I'm able to reproduce the issue, and on other I'm not. I'm looking into whether this is an overall performance issue with the tool or a system-level issue,

For now, using a local redis queue resolves the issue.

[network23]

I have the same error on different Kali rolling machines.

Please install a redis server for better performance.

I have redis-server installed, how to use it?

[ztgrace]

@network23 apt-get install redis-server && /etc/init.d/redis-server start should install the redis-server and start it.

I'm going to try to work on a bug fix for this. My hunch is that the underlying problem is that each fingerprint and scanner instances has a reference to the credential dict and when that writes to the db, it's having to write an excessive amount of data. I'm going to move the creds dict to a global variable so it doesn't get stored along side the scanner or I'll do some more __getstate__/__setstate__ magic.

[network23]

@ztgrace Ah, it uses it automagically. Still the same error, also when I'm using the docker file. You could be right, I tried using "--protocols ssh" and it's working correctly. Even when scanning a local /24.

Traceback (most recent call last): File "./changeme.py", line 6, in core.main() File "/home/X/Desktop/progz/SCAN/changeme/changeme/core.py", line 67, in main check_for_interrupted_scan(config) File "/home/X/Desktop/progz/SCAN/changeme/changeme/core.py", line 394, in check_for_interrupted_scan scanners = FIFOSQLiteQueue(path=".", multithreading=True, name="scanners") File "/home/X/.local/lib/python2.7/site-packages/persistqueue/sqlbase.py", line 70, in init self._init() File "/home/X/.local/lib/python2.7/site-packages/persistqueue/sqlqueue.py", line 40, in _init super(SQLiteQueue, self)._init() File "/home/X/.local/lib/python2.7/site-packages/persistqueue/sqlbase.py", line 85, in _init self._conn.execute(self._sql_create) sqlite3.OperationalError: database is locked

[anshumanbh]

@ztgrace I see the same error:

Traceback (most recent call last):
  File "./changeme-1.0.3/changeme.py", line 6, in <module>
    core.main()
  File "/opt/attack-host/changeme-1.0.3/changeme/core.py", line 70, in main
    s.scan()
  File "/opt/attack-host/changeme-1.0.3/changeme/scan_engine.py", line 65, in scan
    while self.scanners.qsize() > 0:
  File "/usr/local/lib/python2.7/dist-packages/persistqueue/sqlqueue.py", line 84, in qsize
    return self.size
  File "/usr/local/lib/python2.7/dist-packages/persistqueue/sqlqueue.py", line 81, in size
    return self._count()
  File "/usr/local/lib/python2.7/dist-packages/persistqueue/sqlbase.py", line 121, in _count
    row = self._putter.execute(sql).fetchone()
sqlite3.OperationalError: database is locked

I noticed you are using persist-queue package. So, I went to https://github.com/peter-wangxu/persist-queue and noticed that they have a FAQ on their page which states:

persistquest open 2 connections for the db if multithreading=True, the SQLite database is locked until that transaction is committed. The timeout parameter specifies how long the connection should wait for the lock to go away until raising an exception. Default time is 10, increase timeout when creating the queue if above error occurs.

I then tried adding the timeout on line 394 and 395 of core.py to be 100.0 so that it increases from the default of 10.0 but no dice.

I also tried changing multithreading=True to multithreading=False but still kept getting the same error.

[ztgrace]

Yeah, I've experimented with timeouts, threads, etc and have the same issues. What I've noticed is that for a relatively small scan like a /24 the data.db is about 850 MB which is ridiculous. And it's only going to get worse as the number of creds grows.

I'm working on redesigning the fingerprint/scanner pattern so that each class instance doesn't contain a reference to the large creds dict. That way, when the fingerprint/scanner modules get written out to the sqlite db, the records should be considerably smaller.

[ztgrace]

Update: The initial refactor has reduced the size of the data.db from ~850MB to 50MB! However, I'm still getting the dreaded database is locked error.

Rather than beating my head against the wall trying to solve the sqlite issues (seems like it might be related to HAVE_USLEEP compile options which varies by distro) I'll be switching the storage mechanism to :memory: since that was fixed in persist-queue 0.3.1: https://github.com/peter-wangxu/persist-queue/issues/22 and the only persistent option will be redis.

[ztgrace]

This should be resolved by v1.0.4.