DataDog / guarddog

:snake: :mag: GuardDog is a CLI tool to Identify malicious PyPI and npm packages
https://securitylabs.datadoghq.com/articles/guarddog-identify-malicious-pypi-packages/
Apache License 2.0
585 stars 43 forks source link

Improve detection to avoid memory over-usage #400

Closed sobregosodd closed 2 months ago

sobregosodd commented 2 months ago

Rules using <... PATTERN ...> consume a lot of memory, thus it usage should be avoided.

output: {"errors": [{"code": 2, "level": "error", "message": "Error while running rules: \n
                    You are seeing this because the engine was killed.\n\n
                    The most common reason this happens is because it used too much memory.\n                    If your repo is large (~10k files or more), you have three options:\n                    1. Increase the amount of memory available to semgrep\n                    2. Reduce the number of jobs semgrep runs with via `-j <jobs>`. We\n                        recommend using 1 job if you are running out of memory.\n                    3. Scan the repo in parts (contact us for help)\n\n                    Otherwise, it is likely that semgrep is hitting the limit on only some\n                    files. In this case, you can try to set the limit on the amount of memory\n                    semgrep can use on each file with `--max-memory <memory>`. We recommend\n                    lowering this to a limit 70% of the available memory. For CI runs with\n                    interfile analysis, the default max-memory is 5000MB. Without, the default\n                    is unlimited.\n\n                    The last thing you can try if none of these work is to raise the stack\n                    limit with `ulimit -s <limit>`.\n\n                    If you have tried all these steps and still are seeing this error, please\n                    contact us.\n\n                       Error: semgrep-core exited with unexpected output\n\n                       [00.05][\u001b[34mINFO\u001b[0m]: Executed as: /Users/sebastian.obregoso/Library/Caches/pypoetry/virtualenvs/guarddog-PTZDzYc9-py3.10/lib/python3.10/site-packages/semgrep/bin/semgrep-core -json -rules /var/folders/83/v1gvs5x976xdn94tnyk95j8r0000gq/T/tmpau5oqo8e.json -j 10 -targets /var/folders/83/v1gvs5x976xdn94tnyk95j8r0000gq/T/tmpxbi2cjv7 -timeout 5 -timeout_threshold 3 -max_memory 0 -fast\n[00.05][\u001b[34mINFO\u001b[0m]: Version: semgrep-core version: 1.77.0\n\n                    ", "type": "SemgrepError"}], "paths": {"scanned": []}, "results": [], "skipped_rules": [], "version": "1.77.0"}