Using manta-synchttps://github.com/bahamas10/node-manta-sync with a
high concurrency (>= 250), strange behavior starts to happen with manta
client.put() and client.info(). These 2 functions no longer callback
which causes node to terminate early because it has no more events to
process.
Digging into this the bug seems to be in this module, where it may be
the case that the agent socket is opened, data is sent, but no data is
received. In this case currently, no callback will be fired, so
manta client.put/info will have no callback to forward back to the user.
This commit has allowed me to successfully put files with a concurrency
of 300, however anything more than that is still problematic, and that
is a bug I still haven't fully identified. This commit only fixes the
issue of this module not calling back every time.
Running with a concurrency of >= 1000, I start to see ECONNREFUSED
errors that don't appear to be coming from Manta, but instead are coming
from the ssh-agent (I'm using the sshAgentSigner in the node-manta sdk).
I understand that this socket probably wasn't meant to be hammered on like
it currently is, but what baffles me is the signing cache already in
place in the node-manta sdk. Looking through the code it seems like
client.put/info shouldn't hit the socket over and over again like I'm
seeing.
In the meantime, this commit fixes the missing callbacks in the case of
no data, but more research will need to be done to identify the
ECONNREFUSED errors I'm seeing, whether a fix for that belongs in this
module or the node-manta sdk is still in question as well.
$ npm test
> ssh-agent@0.2.1 test /Users/dave/dev/forks/node-ssh-agent
> ./node_modules/.bin/whiskey -t "`find tst -name *.test.js | xargs`"
/Users/dave/dev/forks/node-ssh-agent/tst/agent.test.js
setUp [OK]
test_request_identities [OK]
test_sign [OK]
tearDown [OK]
---------------------------------------------------------------------------------
Ran 4 tests in 0.119s
Successes: 4
Failures: 0
Timeouts: 0
Skipped: 0
PASSED
Using
manta-synchttps://github.com/bahamas10/node-manta-sync with a high concurrency (>= 250), strange behavior starts to happen with manta client.put() and client.info(). These 2 functions no longer callback which causes node to terminate early because it has no more events to process.Digging into this the bug seems to be in this module, where it may be the case that the agent socket is opened, data is sent, but no data is received. In this case currently, no callback will be fired, so manta client.put/info will have no callback to forward back to the user.
This commit has allowed me to successfully put files with a concurrency of 300, however anything more than that is still problematic, and that is a bug I still haven't fully identified. This commit only fixes the issue of this module not calling back every time.
Running with a concurrency of >= 1000, I start to see ECONNREFUSED errors that don't appear to be coming from Manta, but instead are coming from the ssh-agent (I'm using the sshAgentSigner in the node-manta sdk). I understand that this socket probably wasn't meant to be hammered on like it currently is, but what baffles me is the signing cache already in place in the node-manta sdk. Looking through the code it seems like client.put/info shouldn't hit the socket over and over again like I'm seeing.
In the meantime, this commit fixes the missing callbacks in the case of no data, but more research will need to be done to identify the ECONNREFUSED errors I'm seeing, whether a fix for that belongs in this module or the node-manta sdk is still in question as well.