Define Zeno's queuing behavior properly

internetarchive / Zeno

State-of-the-art web crawler 🔱

GNU Affero General Public License v3.0

81 stars 10 forks source link

Define Zeno's queuing behavior properly #115

Closed CorentinB closed 2 months ago

CorentinB commented 3 months ago

As discussed with @equals215, Zeno's behavior in term of queuing should be as followed:

By default: handover + WAL writes batching
As an option: --disable-handover, --whatever (--whatever == a good argument name to ask for more atomicity == not use WAL batch writing)
When using HQ: handover++ (no queuing AT ALL)

equals215 commented 2 months ago

Options and defaults can be done in under a minute. Handover++ as you call it, can either be done on the fly or integrated to #113 in the channeling efforts, as you wish @CorentinB

CorentinB commented 2 months ago

I guess it should be integrated in #113.