peter-wangxu / persist-queue

A thread-safe disk based persistent queue in Python
BSD 3-Clause "New" or "Revised" License
335 stars 50 forks source link

[BUG] "No sqlite3 module found, sqlite3 based queues are not available" during load, despite sqlite3 being present... and also works despite? #202

Closed epicwhale closed 9 months ago

epicwhale commented 9 months ago

I just pip installed 0.8.1 pip install persist-queue with no extras - and in a Jupyternotebook with Python 3.11.7 - when I try:

import persistqueue
q = persistqueue.SQLiteQueue(QUEUE_DATA_DIRECTORY, db_file_name=QUEUE_DB_FILE_NAME, auto_commit=True)

I get an INFO log output in std: No sqlite3 module found, sqlite3 based queues are not available

If I add a 'import sqlite3' at the top of the file, I don't get any error from Python, but this log still appears

Despite this log msg, the queue seems to be creating a sqlite db and working as intended. What's causing this false positive?

epicwhale commented 9 months ago

After cloning this repo, and writing a very rudimentary script, I am able to repro this:

import logging
logging.basicConfig(level=logging.DEBUG)

import persistqueue

q = persistqueue.SQLiteQueue('tmp')

Output

INFO:persistqueue.common:Selected pickle protocol: '4'
INFO:persistqueue:No sqlite3 module found, sqlite3 based queues are not available

It looks like the culprit is the line below which suppresses a "No module named 'dbutils'" with a hardcoded log.

https://github.com/peter-wangxu/persist-queue/blob/65c7fe957a94b5f4186879174e8c6cbb6e4f4aa4/persistqueue/__init__.py#L20

Is there a way to import SQLiteQueue only by circumventing the code in this init.py ?

peter-wangxu commented 9 months ago

Yes, this most likely caused by the missing DBUtils, I will fixit in a coming PR

epicwhale commented 9 months ago

@peter-wangxu hey thanks for looking into this. I was wondering what the proposed fix accomplishes? For users who simply use say the SQLiteQueue and never install the extras, it will still always emit a redundant log each time "INFO:persistqueue:DBUtils may not be installed, install via 'pip install persist-queue[extra]'".

Would a more useful solution be to not import any MySQLQueues' in init.py unless the user explicitly loads that module?