Closed GabrielMioranza closed 3 years ago
Issue-Label Bot is automatically applying the label bug
to this issue, with a confidence of 0.97. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.
Issue Summary
I got a scan failed on every task
Steps to Reproduce
- When I am scanning the domains, this happens
Any other relevant information. For example, why do you consider this a bug and what did you expect to happen instead?
- I have confirmed that this issue can be reproduced as described on a latest version/pull of reNgine: (yes / no) - Yes How i can get the logs and see who happens?
Technical details
Please list out any technical details such as operating environment.
Ubuntu 20;04
met the same problem on ubuntu20.04, have u solved it?
can you please attach the logs?
cd rengine make logs
I use the "oppo.com" as target, scan type is full scan, ubuntu 20.04 virtual machine. The logs:
COMPOSE_DOCKER_CLI_BUILD=1 docker-compose -f docker-compose.yml logs --follow --tail=1000 db web proxy redis celery celery-beat
Attaching to rengine_proxy_1, rengine_celery_1, rengine_web_1, rengine_celery-beat_1, rengine_db_1, rengine_redis_1
celery-beat_1 | Waiting for postgres...
celery-beat_1 | PostgreSQL started
celery-beat_1 | Operations to perform:
celery-beat_1 | Apply all migrations: admin, auth, contenttypes, django_celery_beat, notification, scanEngine, sessions, startScan, targetApp
celery-beat_1 | Running migrations:
celery-beat_1 | Traceback (most recent call last):
celery-beat_1 | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
celery-beat_1 | return self.cursor.execute(sql)
celery-beat_1 | psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "pg_type_typname_nsp_index"
celery-beat_1 | DETAIL: Key (typname, typnamespace)=(auth_permission_id_seq, 2200) already exists.
celery-beat_1 |
celery-beat_1 |
celery-beat_1 | The above exception was the direct cause of the following exception:
celery-beat_1 |
celery-beat_1 | Traceback (most recent call last):
celery-beat_1 | File "manage.py", line 21, in
celery_1 | -------------- celery@03d3c9c207bb v4.4.7 (cliffs)
celery_1 | --- * -----
celery_1 | -- *** ---- Linux-5.4.0-53-generic-x86_64-with 2020-11-23 10:57:17
celery_1 | - --- ---
celery_1 | - ---------- [config]
celery_1 | - ---------- .> app: reNgine:0x7f1a6da56ac0
celery_1 | - ---------- .> transport: redis://redis:6379//
celery_1 | - ---------- .> results: redis://redis:6379/
celery_1 | - --- --- .> concurrency: 4 (prefork)
celery_1 | -- *** ---- .> task events: OFF (enable -E to monitor tasks in this worker)
celery_1 | --- * -----
celery_1 | -------------- [queues]
celery_1 | .> celery exchange=celery(direct) key=celery
celery_1 |
celery_1 |
celery_1 | [tasks]
celery_1 | . reNgine.tasks.doScan
celery_1 | . reNgine.tasks.test_task
celery_1 |
celery_1 | [2020-11-23 10:57:17,427: INFO/MainProcess] Connected to redis://redis:6379//
celery_1 | [2020-11-23 10:57:17,435: INFO/MainProcess] mingle: searching for neighbors
celery_1 | [2020-11-23 10:57:18,456: INFO/MainProcess] mingle: all alone
celery_1 | [2020-11-23 10:57:18,473: WARNING/MainProcess] /usr/local/lib/python3.8/site-packages/celery/fixups/django.py:205: UserWarning: Using settings.DEBUG leads to a memory
celery_1 | leak, never use this setting in production environments!
celery_1 | warnings.warn('''Using settings.DEBUG leads to a memory
celery_1 | [2020-11-23 10:57:18,473: INFO/MainProcess] celery@03d3c9c207bb ready.
web_1 | Waiting for postgres...
web_1 | PostgreSQL started
web_1 | Operations to perform:
web_1 | Apply all migrations: admin, auth, contenttypes, django_celery_beat, notification, scanEngine, sessions, startScan, targetApp
web_1 | Running migrations:
web_1 | Traceback (most recent call last):
web_1 | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
web_1 | return self.cursor.execute(sql)
web_1 | psycopg2.errors.DuplicateTable: relation "django_migrations" already exists
web_1 |
web_1 |
web_1 | The above exception was the direct cause of the following exception:
web_1 |
web_1 | Traceback (most recent call last):
web_1 | File "/usr/local/lib/python3.8/site-packages/django/db/migrations/recorder.py", line 67, in ensure_schema
web_1 | editor.create_model(self.Migration)
web_1 | File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 324, in create_model
web_1 | self.execute(sql, params or None)
web_1 | File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 142, in execute
web_1 | cursor.execute(sql, params)
web_1 | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 68, in execute
web_1 | return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
web_1 | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
web_1 | return executor(sql, params, many, context)
web_1 | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 86, in _execute
web_1 | return self.cursor.execute(sql, params)
web_1 | File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 90, in exit
web_1 | raise dj_exc_value.with_traceback(traceback) from exc_value
web_1 | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
web_1 | return self.cursor.execute(sql)
web_1 | django.db.utils.ProgrammingError: relation "django_migrations" already exists
web_1 |
web_1 |
web_1 | During handling of the above exception, another exception occurred:
web_1 |
web_1 | Traceback (most recent call last):
web_1 | File "manage.py", line 21, in
proxy_1 | 172.21.0.1 - - [23/Nov/2020:10:58:29 +0000] "POST /start_scan/start/1 HTTP/2.0" 302 0 "https://127.0.0.1/start_scan/start/1" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:83.0) Gecko/20100101 Firefox/83.0" "-"
proxy_1 | 172.21.0.1 - - [23/Nov/2020:10:58:29 +0000] "GET /start_scan/history/ HTTP/2.0" 200 28385 "https://127.0.0.1/start_scan/start/1" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:83.0) Gecko/20100101 Firefox/83.0" "-"
celery_1 |
celery1 |
celery1 | ____ | | / () | |_
celery1 | (-< || | ' \ | | ' \/ / -) '_|
celery_1 | //_,|./| ||||_,_|_| v2.4.5
celery_1 |
celery_1 | projectdiscovery.io
celery_1 |
celery_1 | [WRN] Use with caution. You are responsible for your actions
celery_1 | [WRN] Developers assume no liability and are not responsible for any misuse or damage.
celery_1 | [WRN] By using subfinder, you also agree to the terms of the APIs used.
celery_1 |
celery_1 | [INF] Configuration file saved to /root/.config/subfinder/config.yaml
celery_1 | [INF] Enumerating subdomains for oppo.com
celery_1 | [INF] Found 83 subdomains for oppo.com in 30 seconds 5 milliseconds
celery_1 |
celery1 | ____ _ ___
celery_1 | / | _| | | ()| |_|__ /
celery_1 | _ | | | | ' | | / | | | | '|
celery_1 | ) | || | |) | | _ \ |_ _) | |
celery_1 | |/ _,|_./|||/\|____/||
celery_1 |
celery_1 | # Coded By Ahmed Aboul-Ela - @aboul3la
celery_1 |
celery_1 | [-] Enumerating subdomains now for oppo.com
celery_1 | [-] Searching now in Baidu..
celery_1 | [-] Searching now in Yahoo..
celery_1 | [-] Searching now in Google..
celery_1 | [-] Searching now in Bing..
celery_1 | [-] Searching now in Ask..
celery_1 | [-] Searching now in Netcraft..
celery_1 | [-] Searching now in DNSdumpster..
celery_1 | [-] Searching now in Virustotal..
celery_1 | [-] Searching now in ThreatCrowd..
celery_1 | [-] Searching now in SSL Certificates..
celery_1 | [-] Searching now in PassiveDNS..
celery_1 | HTTPSConnectionPool(host='dnsdumpster.com', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fcff54d1e80>: Failed to establish a new connection: [Errno 111] Connection refused'))
celery_1 | Process DNSdumpster-8:
celery_1 | Traceback (most recent call last):
celery_1 | File "/usr/local/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
celery_1 | self.run()
celery_1 | File "/app/tools/Sublist3r/sublist3r.py", line 269, in run
celery_1 | domain_list = self.enumerate()
celery_1 | File "/app/tools/Sublist3r/sublist3r.py", line 649, in enumerate
celery_1 | token = self.get_csrftoken(resp)
celery_1 | File "/app/tools/Sublist3r/sublist3r.py", line 644, in get_csrftoken
celery_1 | token = csrf_regex.findall(resp)[0]
celery_1 | TypeError: expected string or bytes-like object
celery_1 | Process GoogleEnum-4:
celery_1 | Traceback (most recent call last):
celery_1 | File "/usr/local/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
celery_1 | self.run()
celery_1 | File "/app/tools/Sublist3r/sublist3r.py", line 269, in run
celery_1 | domain_list = self.enumerate()
celery_1 | File "/app/tools/Sublist3r/sublist3r.py", line 240, in enumerate
celery_1 | if not self.check_response_errors(resp):
celery_1 | File "/app/tools/Sublist3r/sublist3r.py", line 304, in check_response_errors
celery_1 | if (type(resp) is str or type(resp) is unicode) and 'Our systems have detected unusual traffic' in resp:
celery_1 | NameError: name 'unicode' is not defined
celery_1 | HTTPSConnectionPool(host='www.virustotal.com', port=443): Max retries exceeded with url: /ui/domains/oppo.com/subdomains (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fcff54db490>: Failed to establish a new connection: [Errno 111] Connection refused'))
celery_1 | Process Virustotal-9:
celery_1 | Traceback (most recent call last):
celery_1 | File "/usr/local/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
celery_1 | self.run()
celery_1 | File "/app/tools/Sublist3r/sublist3r.py", line 269, in run
celery_1 | domain_list = self.enumerate()
celery_1 | File "/app/tools/Sublist3r/sublist3r.py", line 703, in enumerate
celery_1 | resp = json.loads(resp)
celery_1 | File "/usr/local/lib/python3.8/json/init.py", line 341, in loads
celery_1 | raise TypeError(f'the JSON object must be str, bytes or bytearray, '
celery_1 | TypeError: the JSON object must be str, bytes or bytearray, not int
celery_1 | HTTPSConnectionPool(host='searchdns.netcraft.com', port=443): Max retries exceeded with url: /?restriction=site+ends+with&host=example.com (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fcff54dc640>: Failed to establish a new connection: [Errno 111] Connection refused'))
celery_1 | Process NetcraftEnum-7:
celery_1 | Traceback (most recent call last):
celery_1 | File "/usr/local/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
celery_1 | self.run()
celery_1 | File "/app/tools/Sublist3r/sublist3r.py", line 269, in run
celery_1 | domain_list = self.enumerate()
celery_1 | File "/app/tools/Sublist3r/sublist3r.py", line 570, in enumerate
celery_1 | cookies = self.get_cookies(resp.headers)
celery_1 | AttributeError: 'NoneType' object has no attribute 'headers'
celery_1 | [-] Saving results to file: oppo.com_2020_11_23_10_58_29/from_sublister.txt
celery_1 | [-] Total Unique Subdomains Found: 19
celery_1 | myoppo.com
celery_1 | www.oppo.com
celery_1 | account.oppo.com
celery_1 | www.account.oppo.com
celery_1 | assorted.downloads.oppo.com
celery_1 | en.oppo.com
celery_1 | d.theme.exapi.oppo.com
celery_1 | expe1.oppo.com
celery_1 | www.expe1.oppo.com
celery_1 | mmevents-hd.oppo.com
celery_1 | mmmail.oppo.com
celery_1 | mx.oppo.com
celery_1 | www.mx.oppo.com
celery_1 | my.oppo.com
celery_1 | myevents-hd.oppo.com
celery_1 | o-doctor.oppo.com
celery_1 | oldcms.oppo.com
celery_1 | ruevents-hd.oppo.com
celery_1 | webexexpe.oppo.com
celery_1 |
celery_1 |
celery_1 | / /
celery1 | / \/ \/ \/ _ \/ // /
celery1 | ////_,/_,/./_,_/ v1
celery_1 |
celery_1 | projectdiscovery.io
celery_1 |
celery_1 | [WRN] Use with caution. You are responsible for your actions
celery_1 | [WRN] Developers assume no liability and are not responsible for any misuse or damage.
celery_1 | [INF] Using host business.oppo.com for enumeration
celery_1 | [INF] Starting scan on host business.oppo.com (106.3.18.159)
celery_1 | [INF] Using host bdev.oppo.com for enumeration
celery_1 | [INF] Starting scan on host bdev.oppo.com (119.147.175.93)
celery_1 | [INF] Using host career.oppo.com for enumeration
celery_1 | [INF] Starting scan on host career.oppo.com (119.147.98.14)
celery_1 | [INF] Using host assorted.downloads.oppo.com for enumeration
celery_1 | [INF] Starting scan on host assorted.downloads.oppo.com (96.7.129.163)
celery_1 | [INF] No ports found on business.oppo.com (106.3.18.159). Host seems down
celery_1 | [INF] Found 1 ports on host bdev.oppo.com (119.147.175.93)
celery_1 | {"host":"bdev.oppo.com","port":25}
celery_1 | [INF] Found 1 ports on host career.oppo.com (119.147.98.14)
celery_1 | {"host":"career.oppo.com","port":110}
celery-beat_1 | [2020-11-23 11:00:21,814: INFO/MainProcess] Writing entries...
celery_1 | [INF] No ports found on assorted.downloads.oppo.com (96.7.129.163). Host seems down
celery_1 | [INF] Using host myoppo.com for enumeration
celery_1 | [INF] Starting scan on host myoppo.com (106.3.18.183)
celery_1 | {"host":"myoppo.com","port":110}
celery_1 | {"host":"myoppo.com","port":25}
celery_1 | [INF] Found 2 ports on host myoppo.com (106.3.18.183)
celery_1 | [INF] Using host prehd.oppo.com for enumeration
celery_1 | [INF] Starting scan on host prehd.oppo.com (106.3.18.243)
celery_1 | [INF] Using host push.oppo.com for enumeration
celery_1 | [INF] Starting scan on host push.oppo.com (36.110.222.129)
celery_1 | {"host":"prehd.oppo.com","port":110}
celery_1 | [INF] No ports found on push.oppo.com (36.110.222.129). Host seems down
celery_1 | [INF] Found 1 ports on host prehd.oppo.com (106.3.18.243)
celery_1 | [INF] Using host yihuan.oppo.com for enumeration
celery_1 | [INF] Starting scan on host yihuan.oppo.com (36.110.222.91)
celery_1 | [INF] Using host yun.oppo.com for enumeration
celery_1 | [INF] Starting scan on host yun.oppo.com (106.3.18.171)
celery_1 | [INF] Found 1 ports on host yihuan.oppo.com (36.110.222.91)
celery_1 | {"host":"yihuan.oppo.com","port":110}
celery_1 | [INF] No ports found on yun.oppo.com (106.3.18.171). Host seems down
celery_1 | flag provided but not defined: -cdn
celery_1 | Usage of httpx:
celery_1 | -H value
celery_1 | Custom Header
celery_1 | -content-length
celery_1 | Content Length
celery_1 | -follow-host-redirects
celery_1 | Only follow redirects on the same host
celery_1 | -follow-redirects
celery_1 | Follow Redirects
celery_1 | -http-proxy string
celery_1 | Http Proxy
celery_1 | -json
celery_1 | JSON Output
celery_1 | -l string
celery_1 | File containing domains
celery_1 | -no-color
celery_1 | No Color
celery_1 | -o string
celery_1 | File to write output to (optional)
celery_1 | -ports value
celery_1 | ports range (nmap syntax: eg 1,2-10,11)
celery_1 | -response-in-json
celery_1 | Server response directly in the tool output (-json only)
celery_1 | -retries int
celery_1 | Number of retries
celery_1 | -silent
celery_1 | Silent mode
celery_1 | -status-code
celery_1 | Extracts Status Code
celery_1 | -store-response
celery_1 | Store Response as domain.txt
celery_1 | -store-response-dir string
celery_1 | Store Response Directory (default current directory) (default ".")
celery_1 | -threads int
celery_1 | Number of threads (default 50)
celery_1 | -timeout int
celery_1 | Timeout in seconds (default 5)
celery_1 | -title
celery_1 | Extracts title
celery_1 | -verbose
celery_1 | Verbose Mode
celery_1 | -version
celery_1 | Show version of httpx
celery_1 | -vhost
celery_1 | Check for VHOSTs
celery_1 | -web-server
celery_1 | Prints out the Server header content
celery_1 | -x string
celery_1 | Request Method (default "GET")
celery_1 | [2020-11-23 11:02:59,567: WARNING/ForkPoolWorker-4] ------------------------------
celery_1 | [2020-11-23 11:02:59,567: WARNING/ForkPoolWorker-4] [Errno 2] No such file or directory: '/app/tools/scan_results/oppo.com_2020_11_23_10_58_29/httpx.json'
celery_1 | [2020-11-23 11:02:59,567: WARNING/ForkPoolWorker-4] ------------------------------
celery_1 | [2020-11-23 11:02:59,578: INFO/ForkPoolWorker-4] Task reNgine.tasks.doScan[7660a41c-6d3c-4faf-b780-bdac4259f6a7] succeeded in 269.65978124699905s: {'status': True}
celery-beat_1 | [2020-11-23 11:03:22,157: INFO/MainProcess] Writing entries...
celery-beat_1 | [2020-11-23 11:06:22,562: INFO/MainProcess] Writing entries...
celery-beat_1 | [2020-11-23 11:09:22,981: INFO/MainProcess] Writing entries...
celery-beat_1 | [2020-11-23 11:12:23,338: INFO/MainProcess] Writing entries...
I have the same issue with an invalid (misconfigured) TLS certificate. There is no way to ignore TLS errors
Issue Summary
I got a scan failed on every task
Steps to Reproduce
Any other relevant information. For example, why do you consider this a bug and what did you expect to happen instead?
Technical details
Please list out any technical details such as operating environment.
Ubuntu 20;04