konnectors / libs

All the libs needed by a cozy v3 konnector
https://docs.cozy.io/en/tutorials/konnector/
MIT License
2 stars 25 forks source link

Manage concurrency limits inside cozy-connector-libs #230

Open edas opened 6 years ago

edas commented 6 years ago

I was recommended not to use Promise.all directly too much because we could hit system's limit on open file descriptors.

The user solution is to use Bluebird's Promise.map with the concurrency parameter.

However, most connectors will only use our functions like requestFactory or saveFiles. It would probably be a good idea to deal with the concurrency internally at least for the functions we manage. Doing so, most developers won't have to even be aware of our system's limits and of the complexity induced.

For requestFactory we could add { pool: {maxSockets: Infinity} } in the default options. Attention, this requires strictSSL to be at true (it is at falsetoday).

For saveFiles we already use Bluebird and only have to add the concurrency parameter.

doubleface commented 6 years ago

This stack overflow post seems to be a good resource about the maxSockets option which I totally missed. And also the official nodejs documentation of course.

I suppose I had some connection error because of an old version of nodejs at that time.

I suppose we could then use Promise.all in most cases (even in saveFiles) and avoid the use of bluebird.