search

jsforce / jsforce

Salesforce API Library for JavaScript applications (both on Node.js and web browser)
https://jsforce.github.io/
MIT License
1.37k stars 534 forks source link

Streaming API back pressure #991

Open dhulke opened 4 years ago

dhulke commented 4 years ago

The Faye package, which the JSForce streaming library uses, will send a /meta/connect message to the server right after receiving the previous message from the server, without waiting for any async work to be done. An application that needs to run async tasks upon receiving these Salesforce Streaming API events, like storing them in a database, has no reliable way to apply backpressure when the number and flow of events surpaces its persistence throughput.

I am facing this very problem today, trying to receive several GBs of Change Data Capture data at once, memory will fill up and the server will start swaping.

To prevent this, I've implemented a change to the Faye package to allow for a subscription mode that waits for the client code to run a callback to send the next /meta/connect message back to the server. Would a pull request implementing this backpressure mechanism make any sense to this project?

SergKam commented 4 years ago

Have the same issue constantly. I don't understand how people run production code without it. This would be great if you could share a patch. Even if it will not be accepted, I would definitely use it.