search

yogeshojha / rengine

reNgine is an automated reconnaissance framework for web applications with a focus on highly configurable streamlined recon process via Engines, recon data correlation and organization, continuous monitoring, backed by a database, and simple yet intuitive User Interface. reNgine makes it easy for penetration testers to gather reconnaissance with minimal configuration and with the help of reNgine's correlation, it just makes recon effortless.
https://yogeshojha.github.io/rengine/
GNU General Public License v3.0
7.39k stars 1.12k forks source link

bug: startScan_command table grows enormous sizes #1324

Open Vos68 opened 1 month ago

Vos68 commented 1 month ago

Is there an existing issue for this?

Current Behavior

Table startScan_command becomes 200Gb + after few scans, and contains result of commands output. Sometimes it looks like 

        +|                               |             |                 |
      |                                                                                                                                                                                 |             | /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00011dfe8 sp=0xc00011dfe0 pc=0x473461

             +|                               |             |                 |
      |                                                                                                                                                                                 |             | github.com/projectdiscovery/nuclei/v3/pkg/templates.(*Parser).LoadTemplate(0x0?, {0xc0012d2e60, 0x44}, {0x25fc7c0?, 0xc000c8ac00}, {0x0, 0x0, 0x0}, {0x3663e50, 0xc000c89660})

Expected Behavior

To keep data organized and store only valuable data.

Steps To Reproduce

I recognized that as problem after upgrading to 2.1.0 I have few hundreds targets and i run scans as usual, but seem all output of commands being recorded and start to be very heavy.

Environment

- reNgine: 2.1.1
- OS: Debian 11
- Python: 3.11
- Docker Engine:  27.0.1
- Docker Compose: v2.28.1
- Browser: Firefox MacOs 128.0 (64-bit)

Anything else?

No response

github-actions[bot] commented 1 month ago

👋 Hi @Vos68, Issues is only for reporting a bug/feature request. Please read documentation before raising an issue https://rengine.wiki For very limited support, questions, and discussions, please join reNgine Discord channel: https://discord.gg/azv6fzhNCE Please include all the requested and relevant information when opening a bug report. Improper reports will be closed without any response.

Vos68 commented 1 month ago

Also in logs 

db-1           |        django.db.utils.IntegrityError: insert or update on table "startScan_command" violates foreign key constraint "startScan_command_activity_id_457a4aaa_fk_startScan"
db-1           |        DETAIL:  Key (activity_id)=(1393) is not present in table "startScan_scanactivity".
db-1           |
db-1           |        ', '4f8834d0-ceb6-45c4-bf35-df1c0cc96876') RETURNING "startScan_scanactivity"."id"