Closed tschaffter closed 3 years ago
I'm almost sure that the issue happens because we sent to many request to the API service in parallel using RxJS forkJoin
. ;-)
The two screenshots below show that the first few requests - tag creation - are fine with little to no time spent in "blocked" request state (red) and most in "waiting" (blue). The second image shows that requests spent most of their time being blocked (red) and ultimately the API service initiates the connection reset issue. I'm now looking at a way to throttle the number of concurrent submission (possibly using this approach).
I found an utility function that behaves like forkJoin
and allows to specify the number of concurrent Observable to process. This solution is implemented in 0f500c1 to post 5 tags at a time. The first screenshot show that no request spend significant time in the "blocked" state. The second screenshot is obtained when settings the concurrency to 20. This leads some requests to spend time in the blocked state. Here the 7th request is the first one that spends time in the blocked state, so 6 concurrent requests would be optimal.
The results shown in this section are obtained when sending requests directly to a Flask API service (development server). See below for the results obtained using uWSGI + Flask (production server).
This time I decided to run the ROCC API service using Docker. This production-oriented approach leverages an uWSGI server placed in front of the Flask app to obtain better performance. The screenshot below shows that up to 20 tags can be pushed concurrently without any of them spending time in the blocked state. Note that slightly different results can be obtained each time the app is reloaded.
By default, I use processes = 1
in server/uwsgi.ini. I test with processes = 5
and definitively obtained less test instances that featured blocked requests. Ideally the value of processes
should not be set larger than the number of CPU cores minus a margin to preserve other important processes (OS, MongoDB, etc.).
@thomasyu888 This ticket provides some insights into ECONNRESET
errors that I encountered while working on the ROCC. This is the same issue that we encountered when seeding the NLP Sandbox Data Node and that we solved by introducing a delay between requests (I implemented both API services in the same way).
Here TypeScript-RxJS is a great solution for performing batch operations like seeding a databases.
When working on #82 , I encounter the following issue:
I now experience
ECONNRESET
errors randomly. Initially after submitting a random number of persons, now also affecting other objects like Organization.See https://nodejs.org/api/errors.html#errors_common_system_errors