Running massdns to resolve subdomains卡住不动了

[Deep0]

2021-01-20 20:24:59,502 [INFOR] MainProcess(8180):MainThread (5456 ) | request.run_request:264 - Found that iqiyi.com has 1542 alive subdomains 2021-01-20 20:27:23,899 [DEBUG] MainProcess(8180):MainThread (5456 ) | iscdn.do_check:67 - Checking cdn 日志就到20点 24点卡住不动好几个小时了。

result.sqlite3-journal一直在OneForAll\results里闪现

[shmilylty]

你敲回车看看

[Ifonly-go2019]

我也一样的情况，回车没用，一直卡住。 OS: macOS

10:55:21,131 [INFOR] request:56 - Generating request urls
10:55:21,132 [INFOR] request:217 - Requesting urls in bulk
Request Progress:  98%|██████████████████████▌| 474/483 [00:16<00:00, 13.40it/s]10:55:48,747 [INFOR] request:264 - Found that xxx has 837 alive subdomains
Request Progress: 484it [00:27, 17.45it/s]

Ctrl C 停止出现如下报错

^CTraceback (most recent call last):
  File "oneforall.py", line 273, in <module>
    fire.Fire(OneForAll)
  File "/usr/local/lib/python3.8/site-packages/fire/core.py", line 138, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/usr/local/lib/python3.8/site-packages/fire/core.py", line 466, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/usr/local/lib/python3.8/site-packages/fire/core.py", line 675, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "oneforall.py", line 250, in run
    self.main()
  File "oneforall.py", line 211, in main
    enrich.run()
  File "/Users/m0nk3y/tools/info/OneForAll/modules/enrich.py", line 72, in run
    self.save_db(data)
  File "/Users/m0nk3y/tools/info/OneForAll/modules/enrich.py", line 65, in save_db
    db.update_data_by_url(self.domain, info, url)
  File "/Users/m0nk3y/tools/info/OneForAll/common/database.py", line 271, in update_data_by_url
    return self.query(sql)
  File "/Users/m0nk3y/tools/info/OneForAll/common/database.py", line 38, in query
    results = self.conn.query(sql)
  File "/Users/m0nk3y/tools/info/OneForAll/common/records.py", line 335, in query
    cursor = self._conn.execute(text(query), **params)  # TODO: PARAMS GO HERE
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement
    ret = self._execute_context(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context
    self._handle_dbapi_exception(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1514, in _handle_dbapi_exception
    util.raise_(exc_info[1], with_traceback=exc_info[2])
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
    cursor.execute(statement, parameters)
KeyboardInterrupt

em… 是等了很久，也不知道是不是正常的，感觉不应该卡那么久（

[shmilylty]

@Deep0 @Ifonly-go2019 这个地方看了一下 ，住的原因是因为在将结果写入数据库，由于目前的代码有多个线程请求子域，但是只有一个线程写入数据库，所以导致请求完所有子域后还有一堆结果没写入数据库，sqlite3貌似不支持多线程写入，后面看看怎么优化一下。

[austfish]

这个问题该怎么避免，有快速的解决办法么？

[shmilylty]

这个问题该怎么避免，有快速的解决办法么？

避免的话，在子域请求时可以尝试指定小范围端口。