search

minvws / nl-kat-coordination

Repo nl-kat-coordination for minvws
European Union Public License 1.2
123 stars 55 forks source link

Nuclei boefje causes github unauthenticated ratelimit for templates #2565

Open tobiasBDO opened 7 months ago

tobiasBDO commented 7 months ago

Please add bug, the name of any relevant modules (e.g. rocky), and any other relevant labels to your issue.

Describe the bug When running Nuclei boefjes the container tries to get the templates from github, if this happens too many times consecutively this causes a ratelimit and thereby "crashing" the boefje.

To Reproduce Steps to reproduce the behavior:

  1. Enable nuclei takeover/panel/cve boefje
  2. Add number of scannable hosts
  3. Wait for boefjes to be run

Expected behavior I expect the boefjes not to crash or for the docker container to have templates.

OpenKAT version Release v1.14.1

Additional context

boefje logs:

nl-kat-coordination_boefje_1               | [2024-02-27 12:03:30 +0000] [25] [INFO] [job_handler] Starting boefje nuclei-cve[99d8b8c3-4e23-440e-9011-c85d298858d6]
nl-kat-coordination_boefje_1               | [2024-02-27 12:03:30 +0000] [25] [INFO] [local] Running local boefje plugin
nl-kat-coordination_boefje_1               | [2024-02-27 12:03:31 +0000] [25] [ERROR] [job_handler] Error running boefje nuclei-cve[99d8b8c3-4e23-440e-9011-c85d298858d6]
nl-kat-coordination_boefje_1               | Traceback (most recent call last):
nl-kat-coordination_boefje_1               |   File "/app/boefjes/boefjes/local.py", line 53, in run
nl-kat-coordination_boefje_1               |     return boefje_resource.module.run(boefje_meta)
nl-kat-coordination_boefje_1               |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nl-kat-coordination_boefje_1               |   File "/app/boefjes/boefjes/plugins/kat_nuclei_cve/main.py", line 26, in run
nl-kat-coordination_boefje_1               |     output = client.containers.run(
nl-kat-coordination_boefje_1               |              ^^^^^^^^^^^^^^^^^^^^^^
nl-kat-coordination_boefje_1               |   File "/usr/local/lib/python3.11/site-packages/docker/models/containers.py", line 887, in run
nl-kat-coordination_boefje_1               |     raise ContainerError(
nl-kat-coordination_boefje_1               | docker.errors.ContainerError: Command '['-t', '/root/nuclei-templates/http/cves/', '-u', 'ictguru.nl:443', '-jsonl']' in image 'projectdiscovery/nuclei:v2.9.4' returned non-zero exit status 1: b"\n                     __     _\n   ____  __  _______/ /__  (_)\n  / __ \\/ / / / ___/ / _ \\/ /\n / / / / /_/ / /__/ /  __/ /\n/_/ /_/\\__,_/\\___/_/\\___/_/   v2.9.4\n\n\t\tprojectdiscovery.io\n\n[\x1b[34mINF\x1b[0m] nuclei-templates are not installed, installing...\n[\x1b[31mERR\x1b[0m] Could not find template '/root/nuclei-templates/http/cves/': could not find file: stat /root/nuclei-templates/http/cves: no such file or directory\n[\x1b[33mWRN\x1b[0m] Found 1 template[s] loaded with deprecated paths, update before v2.9.5 for continued support.\n[\x1b[34mINF\x1b[0m] Current nuclei version: v2.9.4 (\x1b[91moutdated\x1b[0m)\n[\x1b[34mINF\x1b[0m] Current nuclei-templates version:  (\x1b[91moutdated\x1b[0m)\n[\x1b[34mINF\x1b[0m] Targets loaded for current scan: 1\n[\x1b[34mINF\x1b[0m] No results found. Better luck next time!\n[\x1b[1;31mFTL\x1b[0m] Could not run nuclei: no valid templates were found\n"
nl-kat-coordination_boefje_1               | 
nl-kat-coordination_boefje_1               | The above exception was the direct cause of the following exception:
nl-kat-coordination_boefje_1               | 
nl-kat-coordination_boefje_1               | Traceback (most recent call last):
nl-kat-coordination_boefje_1               |   File "/app/boefjes/boefjes/job_handler.py", line 150, in handle
nl-kat-coordination_boefje_1               |     boefje_results = self.job_runner.run(boefje_meta, boefje_meta.environment)
nl-kat-coordination_boefje_1               |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nl-kat-coordination_boefje_1               |   File "/app/boefjes/boefjes/local.py", line 55, in run
nl-kat-coordination_boefje_1               |     raise JobRuntimeError("Boefje failed") from e
nl-kat-coordination_boefje_1               | boefjes.runtime_interfaces.JobRuntimeError: Boefje failed

Manual test:

docker run projectdiscovery/nuclei:v2.9.4 -u ictguru.nl -v

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v2.9.4

        projectdiscovery.io

[INF] nuclei-templates are not installed, installing...
[WRN] failed to install nuclei templates: [:RUNTIME] failed to install templates at /root/nuclei-templates <- [:RUNTIME] failed to install templates at /root/nuclei-templates <- [:RUNTIME] hit github ratelimit while downloading latest release <- GET https://api.github.com/repos/projectdiscovery/nuclei-templates/releases/latest: 403 API rate limit exceeded for 152.42.132.111. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.) [rate reset in 8m30s]
[WRN] failed to update nuclei templates: [:RUNTIME] failed to install templates at /root/nuclei-templates <- [:RUNTIME] hit github ratelimit while downloading latest release <- GET https://api.github.com/repos/projectdiscovery/nuclei-templates/releases/latest: 403 API rate limit exceeded for 152.42.132.111. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.) [rate reset in 8m30s]
[ERR] Could not find template '/root/nuclei-templates': no templates found in path /root/nuclei-templates
[INF] Current nuclei version: v2.9.4 (outdated)
[INF] Current nuclei-templates version:  (outdated)
[INF] Targets loaded for current scan: 1
[INF] No results found. Better luck next time!
[FTL] Could not run nuclei: no valid templates were found
tobiasBDO commented 7 months ago

I modified some of the code in order for the nuclei boefje to use a template directory on my host system, this was a temporary way to bypass ratelimiting. The problem is that at every container run it tries to pull templates from github and since "The primary rate limit for unauthenticated requests is 60 requests per hour."^1 it fills up rather quickly

zcrt commented 6 months ago

Somewhat related to https://github.com/minvws/nl-kat-coordination/issues/152. Maybe KAT could include multiple databases (CVE, CWE, nuclei templates, etc.) that can be updated completely independent (independent of boefjes that use these local template and/or enrichment databases and independent of the kat-installation-version to stay up-to-date).

underdarknl commented 6 months ago

Yes, we need some sort of caching mechanism for stuff like this.