shmilylty / OneForAll

OneForAll是一款功能强大的子域收集工具
GNU General Public License v3.0
8.08k stars 1.28k forks source link

'NoneType' object has no attribute 'scalar' #389

Open JackeyLov5 opened 1 month ago

JackeyLov5 commented 1 month ago

是否使用了最新代码 是或否(如果不是的话尝试克隆最新的代码再跑一下!)

Bug描述 'NoneType' object has no attribute 'scalar'

sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 123150260609024 and this is thread id 123149589028864.

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='1ec687a4db284800.static-cloudflareinsights-com-s.webvpn.gnnu.edu.cn', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x109f382e0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))

urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x109f382e0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known

运行环境

如何复现 python3 oneforall.py --targets ./domains.txt run

domains.txt 内容为: gnnu.edu.cn gnnu.cn

复现命令(必写) python3 oneforall.py --targets ./domains.txt run Request Progress: 9%|█▉ | 270/2854 [00:00<00:05, 476.78it/s]12:33:05,668 [ERROR] database:40 - ('(sqlite3.OperationalError) unable to open database file',) Exception in thread SaveThread: Traceback (most recent call last): File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/sqlalchemy/engine/default.py", line 609, in do_execute sqlite3.OperationalError: no such table: gnnu_edu_cn

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/threading.py", line 973, in _bootstrap_inner File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/threading.py", line 910, in run File "/Users/ookk/Desktop/tools/信息收集/OneForAll/common/request.py", line 209, in save db.insert_table(name, new_info) File "/Users/ookk/Desktop/tools/信息收集/OneForAll/common/database.py", line 91, in insert_table File "/Users/ookk/Desktop/tools/信息收集/OneForAll/common/records.py", line 335, in query File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/sqlalchemy/engine/base.py", line 1011, in execute return meth(self, multiparams, params) File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/sqlalchemy/engine/base.py", line 1510, in _handle_dbapiexception File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/sqlalchemy/util/compat.py", line 182, in raise File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/sqlalchemy/engine/default.py", line 609, in do_execute cursor.execute(statement, parameters) sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: gnnu_edu_cn [SQL: insert into "gnnu_edu_cn" (id, alive, resolve, request, url, subdomain, port, level,cname, ip, public, cdn, status, reason, title, banner, header,history, response, ip_times, cname_times, ttl, cidr, asn, org,addr, isp, resolver, module, source, elapse, find) values (?, ?, ?, ?, ?,?, ?, ?, ?, ?, ?, ?,?, ?, ?, ?, ?, ?, ?,?, ?, ?, ?, ?, ?, ?, ?,?, ?, ?, ?, ?)] [parameters: (None, 0, 1, 0, 'https://dmt.gnnu.edu.cn', 'dmt.gnnu.edu.cn', 443, 1, 'dmt.gnnu.edu.cn', '218.64.216.247', None, None, None, '(MaxRetryError("HTTPSConnectionPool(host=\'dmt.gnnu.edu.cn\', port=443): Max retries exceeded with url: / (Caused by NewConnectionError(\'<urllib3.connection.HTTPSConnection object at 0x109774d60>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known\'))"),)', None, None, None, None, None, None, None, '3600', None, None, None, None, None, '1.0.0.19:53', 'Dataset', 'SecurityTrailsAPIQuery', 2.1, 150)] (Background on this error at: http://sqlalche.me/e/13/e3q8) Request Progress: 22%|████▌ | 617/2854 [00:03<00:16, 135.04it/s]Exception during reset or similar Traceback (most recent call last): File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/urllib3/connection.py", line 174, in _new_conn conn = connection.create_connection( File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/urllib3/util/connection.py", line 72, in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/socket.py", line 953, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): socket.gaierror: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/urllib3/connectionpool.py", line 703, in urlopen httplib_response = self._make_request( File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/urllib3/connectionpool.py", line 386, in _make_request self._validate_conn(conn) File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn conn.connect() File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/urllib3/connection.py", line 358, in connect self.sock = conn = self._new_conn() File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/urllib3/connection.py", line 186, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x109f382e0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/urllib3/connectionpool.py", line 787, in urlopen File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/urllib3/util/retry.py", line 592, in increment urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='1ec687a4db284800.static-cloudflareinsights-com-s.webvpn.gnnu.edu.cn', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x109f382e0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/sqlalchemy/pool/base.py", line 697, in _finalize_fairy fairy._reset(pool) File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/sqlalchemy/pool/base.py", line 893, in _reset pool._dialect.do_rollback(self) File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/sqlalchemy/engine/default.py", line 559, in do_rollback sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 123150260609024 and this is thread id 123149589028864. Request Progress: 22%|████▋ | 640/2854 [00:03<00:14, 155.34it/s]Exception closing connection <sqlite3.Connection object at 0x108d5c3f0> Traceback (most recent call last): File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/urllib3/connection.py", line 174, in _new_conn conn = connection.create_connection( File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/urllib3/util/connection.py", line 72, in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/socket.py", line 953, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): socket.gaierror: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/urllib3/connectionpool.py", line 703, in urlopen httplib_response = self._make_request( File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/urllib3/connectionpool.py", line 386, in _make_request self._validate_conn(conn) File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn conn.connect() File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/urllib3/connection.py", line 358, in connect self.sock = conn = self._new_conn() File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/urllib3/connection.py", line 186, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x109f382e0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/urllib3/connectionpool.py", line 787, in urlopen retries = retries.increment( File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/urllib3/util/retry.py", line 592, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='1ec687a4db284800.static-cloudflareinsights-com-s.webvpn.gnnu.edu.cn', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x109f382e0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/sqlalchemy/pool/base.py", line 697, in _finalize_fairy fairy._reset(pool) File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/sqlalchemy/pool/base.py", line 893, in _reset pool._dialect.do_rollback(self) File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/sqlalchemy/engine/default.py", line 559, in do_rollback dbapi_connection.rollback() sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 123150260609024 and this is thread id 123149589028864.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/sqlalchemy/pool/base.py", line 270, in _close_connection self._dialect.do_close(connection) File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/sqlalchemy/engine/default.py", line 565, in do_close dbapi_connection.close() sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 123150260609024 and this is thread id 123149589028864. Request Progress: 2855it [00:49, 58.11it/s] 12:33:54,270 [ERROR] database:40 - ('(sqlite3.OperationalError) no such table: gnnu_edu_cn',) Traceback (most recent call last): File "/Users/ookk/Desktop/tools/信息收集/OneForAll/oneforall.py", line 275, in fire.Fire(OneForAll) File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/fire/core.py", line 141, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/fire/core.py", line 466, in _Fire component, remaining_args = _CallAndUpdateTrace( File "/Users/ookk/Library/Python/3.9/lib/python/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) File "/Users/ookk/Desktop/tools/信息收集/OneForAll/oneforall.py", line 252, in run self.main() File "/Users/ookk/Desktop/tools/信息收集/OneForAll/oneforall.py", line 198, in main request.run_request(self.domain, self.data, self.port) File "/Users/ookk/Desktop/tools/信息收集/OneForAll/common/request.py", line 263, in run_request count = utils.count_alive(domain) File "/Users/ookk/Desktop/tools/信息收集/OneForAll/common/utils.py", line 425, in count_alive count = result.scalar() AttributeError: 'NoneType' object has no attribute 'scalar'

预期结果

实际结果 清晰而简洁的实际结果描述(选写,如出现什么错误)

屏幕截图 完整OneForAll执行流程截图(建议上传)

日志上传 上传oneforall.log日志文件(复杂问题建议上传)

其他补充 关于bug的其他一些补充说明

JackeyLov5 commented 1 month ago

找到了之前师傅说的 用主域名 可以。但是hw中的 edu.cn 和 gov.cn怎么办呢