implydata / plywood

A toolkit for querying and interacting with Big Data
https://plywood.imply.io
Apache License 2.0
384 stars 61 forks source link

concurrent limit requester getting stuck #285

Open cedrics opened 1 year ago

cedrics commented 1 year ago

We recently started observing plywood seemingly getting stuck. After some investigation we realized the concurrent limit requester is getting stuck and from its perspective the number of requests didn't decrease anymore.

I am not very familiar with node's streaming API but we patched the requester to listen to the close event instead of end & error and haven't observed a similar issue since, however. I am not a 100% sure if this actually fixed the issue as it only popped up sporadically or if there any drawbacks of using close. The node documentation states that the end event will not be triggered unless all data is read from the stream and my assumption is that we are somehow running into that edge case