search

webrecorder / browsertrix

Browsertrix is the hosted, high-fidelity, browser-based crawling service from Webrecorder designed to make web archiving easier and more accessible for all!
https://webrecorder.net/browsertrix
GNU Affero General Public License v3.0
201 stars 35 forks source link

[Feature]: Crawler cpu_limit #2094

Open prestonvanloon opened 1 month ago

prestonvanloon commented 1 month ago

What change would you like to see?

I would like to set resources.cpu_limit on the crawler pods so that the cluster is not overwhelmed.

Context

There is no way to set cpu_limit for the crawler pods.

ikreymer commented 1 month ago

This is generally not needed, since crawler pods all have a minimum CPU request - the remaining CPU is distributed amongst the available pods. For more information why CPU requests are good, but CPU limits are bad, see: https://home.robusta.dev/blog/stop-using-cpu-limits