metafacture / metafacture-core

Core package of the Metafacture tool suite for metadata processing.
https://metafacture.org
Apache License 2.0
72 stars 34 forks source link

Add a flux command for waiting a specific time #495

Open fsteeg opened 1 year ago

fsteeg commented 1 year ago

Use case is a workflow where we use open-http to POST records to an API. When working with large input data, we'd like to wait a short amount of time between each post to avoid creating too much load on the server. While this could be done with an additional option for open-http, perhaps a generic wait command, with the option to set the time would be the best approach?

blackwinter commented 1 year ago

Does wait imply waiting for something? (Like wait-for-inputs for instance.)

Maybe sleep would be better? (Could also be a Fix function; see Catmandu.)

TobiasNx commented 1 month ago

I would upvote this. It became relevant to me too when trying to scrape a website with to many requests but without getting banned but getting shortly blocked for the frequency of the requests.