search

my8100 / scrapydweb

Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. DEMO :point_right:
https://github.com/my8100/files
GNU General Public License v3.0
3.15k stars 562 forks source link

High Memory Usage with ScrapydWeb #236

Open vikhilrestaverse opened 2 months ago

vikhilrestaverse commented 2 months ago

High Memory Usage with ScrapydWeb

I've observed an issue where ScrapydWeb causes excessive memory usage when running alongside Scrapyd. On an EC2 instance with 8GB of RAM and 2 vCPUs, Scrapyd alone uses around 2GB of memory, which is expected. However, when ScrapydWeb is started, the memory usage spikes to 100%, leading to the instance becoming unresponsive and eventually stopping.

Expected behavior ScrapydWeb should operate without causing such a drastic increase in memory usage, allowing the system to remain stable.

Actual behaviour When ScrapydWeb is running, the memory usage increases significantly, leading to the system becoming unresponsive.

Environment: Instance: AWS EC2, 8GB RAM, 2 vCPUs Scrapy Version: 2.11.2 Scrapyd Version: 1.4.3 ScrapydWeb Version: 1.5.0 Operating System: Linux Arm

Additional Information: This issue occurs consistently with ScrapydWeb. The instance is running minimal other processes, so the memory spike is attributable to ScrapydWeb.

*Request for Assistance: Any guidance on resolving this memory usage issue or optimizing ScrapydWeb would be greatly appreciated.

vikhilrestaverse commented 2 months ago

@my8100 Can you please giude on this ?