oneforall内存占用过高

leveryd commented 1 year ago

https://github.com/leveryd-asm/asm/blob/94349fd8c5/templates/argo-workflow-template/level1/subdomain.yaml

python3 oneforall.py --target {{inputs.parameters.domain}} --dns false run

比如输入域名是 qq.com 时，程序运行过程中，内存占用就已经到5G+

leveryd commented 1 year ago

奇怪的是同一个镜像，如下执行后

[root@192 ~]# docker run -ti --entrypoint sh --rm shmilylty/oneforall:version-0.4.3
/OneForAll # python3 oneforall.py --target qq.com run

top -p pid 观察内存占用，发现内存占用1G不到，很小。

按照以上步骤，第二次测试，又发现内存占用峰值也会到6G，很大

leveryd commented 1 year ago

尝试用最新的v0.4.5版本测试，结果执行报错

/tmp/OneForAll-0.4.5 # python3 oneforall.py --target qq.com run

OneForAll is a powerful subdomain integration tool
             ___             _ _
 ___ ___ ___|  _|___ ___ ___| | | {v0.4.5 #dev}
| . |   | -_|  _| . |  _| .'| | |
|___|_|_|___|_| |___|_| |__,|_|_| git.io/fjHT1

OneForAll is under development, please update before each use!

[*] Starting OneForAll @ 2023-02-22 10:58:16

10:58:16,160 [INFOR] utils:523 - Checking dependent environment
10:58:16,160 [INFOR] utils:535 - Checking network environment
10:58:44,401 [ERROR] utils:509 - (ReadTimeoutError("HTTPConnectionPool(host='39.102.194.95', port=80): Read timed out. (read timeout=27)"),)
10:58:44,401 [ALERT] utils:510 - Unable to access Internet, retrying...
10:58:48,871 [ALERT] utils:540 - Please check your network environment.
10:58:48,871 [INFOR] oneforall:243 - Start running OneForAll
10:58:48,873 [INFOR] oneforall:248 - Got 1 domains
10:58:48,890 [ALERT] oneforall:153 - Because it cannot access the Internet, OneForAll will not execute the subdomain collection module!
10:58:54,062 [INFOR] module:63 - BruteSRV module took 5.2 seconds found 1 subdomains
10:58:54,071 [INFOR] brute:461 - Start running Brute module
10:58:54,071 [INFOR] utils:535 - Checking network environment
10:59:08,190 [ALERT] utils:540 - Please check your network environment.
10:59:08,191 [INFOR] brute:411 - Blasting qq.com
10:59:08,191 [INFOR] brute:119 - Querying NS records of qq.com
10:59:08,201 [INFOR] brute:129 - qq.com's authoritative name server is ['ns3.qq.com.', 'ns1.qq.com.', 'ns4.qq.com.', 'ns2.qq.com.']
10:59:08,201 [INFOR] brute:99 - Querying A record from authoritative name server: ['ns3.qq.com.', 'ns1.qq.com.', 'ns4.qq.com.', 'ns2.qq.com.']
10:59:08,224 [INFOR] brute:114 - Authoritative name server A record result: ['112.60.1.69', '203.205.195.94', '183.192.164.81', '101.89.19.165', '203.205.220.251', '157.255.246.101', '183.36.112.46', '125.39.46.125', '203.205.195.104', '59.36.132.142', '218.68.91.143', '58.144.154.100', '123.151.66.78', '211.100.32.218', '203.205.249.143', '121.51.160.100', '203.205.177.41']
10:59:08,224 [INFOR] brute:348 - Generating dictionary for qq.com
10:59:08,374 [ALERT] utils:686 - Please check whether zhaoshang.qq.com is correct or not
10:59:08,378 [INFOR] brute:366 - Dictionary size: 95247
10:59:08,407 [INFOR] brute:442 - Running massdns to brute subdomains
/bin/sh: /tmp/OneForAll-0.4.5/thirdparty/massdns/massdns_linux_x86_64: not found
Traceback (most recent call last):
  File "oneforall.py", line 277, in <module>
10:59:08,411 [INFOR] brute:197 - Counting IP cname appear times
    fire.Fire(OneForAll)
  File "/usr/local/lib/python3.8/site-packages/fire/core.py", line 138, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/usr/local/lib/python3.8/site-packages/fire/core.py", line 463, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/usr/local/lib/python3.8/site-packages/fire/core.py", line 672, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "oneforall.py", line 254, in run
    self.main()
  File "oneforall.py", line 171, in main
    brute.run()
  File "/tmp/OneForAll-0.4.5/brute.py", line 473, in run
    valid_subdomains = self.main(self.domain)
  File "/tmp/OneForAll-0.4.5/brute.py", line 446, in main
    appear_times = stat_appear_times(output_path)
  File "/tmp/OneForAll-0.4.5/brute.py", line 200, in stat_appear_times
    with open(result_path) as fd:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/OneForAll-0.4.5/results/temp/resolved_result_qq.com_20230222_105908.json'

leveryd-asm / asm

oneforall内存占用过高 #14