search

Informatievlaanderen / VSDS-Linked-Data-Interactions

https://informatievlaanderen.github.io/VSDS-Linked-Data-Interactions/
European Union Public License 1.2
4 stars 6 forks source link

Add configuration of concurrency/ping speed of the LDES Client #178

Closed xdxxxdx closed 1 year ago

xdxxxdx commented 1 year ago

As lots of data source has a rate-limiting configured on their server.

To avoid receiving 429 responses all the time, I found it interesting to be able to configure the client's concurrency. For example, if we know the restrictions of a certain API we could say "do 1 page at a time" or "no more than X requests per 10 minutes"

https://github.com/Informatievlaanderen/VSDS-Linked-Data-Interactions/tree/main/ldi-orchestrator/ldio-connectors/ldio-ldes-client

sandervd commented 1 year ago

It would be interesting to make the scheduling strategy pluggable... (exponential back-off, simple time between requests,...)

Tomvbe commented 1 year ago

It would be interesting to make the scheduling strategy pluggable... (exponential back-off, simple time between requests,...)

Currently when retry is enabled (soon it will be enabled by default), it comes with exponential random back-off by default. The retry http request executor expects a resilience4j config which allows us to easily add more options such as a simple time interval if this would be required in future use cases.

xdxxxdx commented 1 year ago

To be tested with Sprint 28

xdxxxdx commented 1 year ago

validated on ldes/ldi-orchestrator:1.0.0-SNAPSHOT