search

ovh / beamium

Prometheus to Warp10 metrics forwarder
Other
84 stars 21 forks source link

feat(batch): use batch size to split scraped sources #73

Closed FlorentinDUBOIS closed 6 years ago

FlorentinDUBOIS commented 6 years ago
StevenLeRoux commented 6 years ago

If batch size is used directly from the scraping layer, do you see any case where it's still accurate to have it on the routing layer?

FlorentinDUBOIS commented 6 years ago

Hi!

There is some cases where it is still accurate to keep this batching mode in the router.

For example, when scraped sources provide content which is inferior to the batch size or during the injection of files in sources folder.