search

crawlab-team / crawlab

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
https://www.crawlab.cn
BSD 3-Clause "New" or "Revised" License
11.38k stars 1.8k forks source link

Master node increases ram memory day by day. #1267

Closed crissvilleda closed 1 year ago

crissvilleda commented 1 year ago

Describe the bug I successfully implemented the Crawlab software utilizing the power of Kubernetes, consisting of a single Node Master and three Worker nodes. On a daily basis, I undertake the rigorous task of executing a total of 23 tasks. Unfortunately, I encountered an issue where the Node Master would crash. Upon conducting a thorough investigation, I discovered that the memory RAM usage of the Node Master consistently increases after executing all tasks each day.

Expected behavior The expected behavior of the system is for the Node Master to smoothly carry out its responsibilities without any crashes or system failures. Additionally, it is also expected for the memory RAM usage of the Node Master to remain stable and not increase significantly after executing all tasks each day.

Screenshots The memory usage of the first day image The memory usage in the second day image

This goes on and on until it crash, the Node Master restarted and I lost the files of the scrapy bot.

tikazyq commented 1 year ago

Which version you are using?

crissvilleda commented 1 year ago

Hi, thanks to respond I am using the crawlab's version V0.6.0

ma-pony commented 1 year ago

Hi, thanks to respond I am using the crawlab's version V0.6.0

It is recommended that you upgrade to 0.6.1, which fixes some memory overflow bugs

tikazyq commented 1 year ago

As @ma-pony mentioned, v0.6.1 fixed memory leaking issue. You can upgrade and see if this issue persist.