mwmbl / crawler-extension

A browser extension that can be installed by volunteers to participate in mwmbl distributed crawling.
GNU Affero General Public License v3.0
21 stars 2 forks source link

Add an option to pause crawling #25

Open fawaf opened 2 years ago

fawaf commented 2 years ago

add option to disable crawling (e.g. the extension) in case of slowof cpu, slow network, other reason, etc.

daoudclarke commented 2 years ago

Also requested in a review: https://addons.mozilla.org/en-GB/android/addon/mwmbl-web-crawler/reviews/1853830/

daoudclarke commented 2 years ago

Since it is likely the user will forget to turn it back on (I know I will) I suggest we allow pausing, but not disabling. We could have two buttons:

  1. Pause for +1h. Pressing the button displays the message "Paused until xx:xx". Pressing repeatedly will increment the number of hours it's paused for.
  2. Resume. Pressing this button removes the message and resumes crawling.

If they want to turn it off completely they can do that through the Firefox extension UI.

daoudclarke commented 2 years ago

Behaviour for batches in progress: we need to check if pause has been pressed in the middle of processing a batch. If that is the case, then we should send the batch in its current state (even if there are just a few pages crawled) and stop crawling.

fawaf commented 2 years ago

err, i meant pausing mainly yeah, heh. maybe allow for a configurable option insead of just a "pause for +1 hour" button? a dropdown or even a direct textbox

daoudclarke commented 2 years ago

Personally I'd prefer to keep it simple, but I'll leave it to whoever implements it to decide.

kristianpaul commented 2 years ago

yeah, please add this, CPU was killing my laptop (i5,16G) recently and had to disable extension manually… Sorry guys

xplosionmind commented 1 year ago

It would be great to implement a simple keyboard shortcut to toggle crawling on and off (for exampbe cmd + shift + M).

benob commented 11 months ago

Another use case: disable on metered connections.