Closed patmagee closed 9 years ago
patmagee
commented
9 years ago I think I may have figured this error out. I think there was a leak between the. Buffer and the number off documents being inserted each iteration, resulting in an array within node essentially the same size as the file itself. Will know in the morning!
ronammar
commented
9 years ago Ok, we need to make sure that no VCF lines are being dropped during these insertion processes. Let’s double check the VCF count post-insertion every time we’re done with a file before we delete it. As a safeguard.
patmagee
commented
9 years ago
After uploading 3 WGS variant call filles the server appeared to crash. The issues appears that somewhere there is blocking code stopping the server performing properly. It is unable to send and receive requests, additionally the parse process stops uploading information to the mongoDB server, or it does so, at an exceptionally slow rate.
The processing of files and handling requests with the server may be too much for it to handle on a single thread at this point, and despite it being Async it is rendered useless.
One possible fix for this could be a side loaded Annotation server.
The annotation server would run along side the main node web server, but would operate under different, resources.