search

TREEcg / event-stream-client

Deprecated! Use the rdf-connect/ldes-client instead
https://github.com/rdf-connect/ldes-client
Other
14 stars 9 forks source link

Take advantage of Node's built-in streaming backpressure #77

Open erikap opened 6 months ago

erikap commented 6 months ago

This PR contains an attempt to take advantage of Node's built-in backpressure in streams (i.e. buffering in case of a slower consumer) instead of trying to manage it ourselves. It also includes an exponential backoff mechanism to retry failing requests.

This PR should not be merged as such. We've removed some code (like for example the paused flag) which may be considered a breaking change (although it looks like an internal to us) and didn't extensively test all different kinds of representations. Given that a new implementation is being worked on in https://github.com/TREEcg/ldes-client, it does not seem appropriate to us to further clean up and elaborate on this PR, but we wanted to indicate what our issues have been with this client so that they can possibly be taken into account in the new implementation.

Co-author: @madnifcent

pietercolpaert commented 5 months ago

@ajuvercr Can you take a look to this at your earliest convenience and analyse whether there’s backpressure implemented in the new ldes-client?

@erikap Is the code in the PR functional and would you want us to release the last package of the deprecated event-stream-client with these changes?

ajuvercr commented 5 months ago

The new client certainly has backpressure, it uses the mdn streams interface and pauses everything after a member is found and resumes when a new member is expected.

If you need any other features in the new client that prevents you from changing, please let us know! @erikap @madnificent