search

TritonDataCenter / node-manta

Node.js SDK for Manta
75 stars 54 forks source link

MPU-related tests should detect if MPU is supported #319

Closed jordanhendricks closed 7 years ago

jordanhendricks commented 7 years ago

If you run the MPU-related tests (mmpu and client method tests), you will fail a bunch of the test suite.

For reference, here is what the test output looks like if run against a Manta without MPU enabled:

$ make test
npm install
unset MANTA_DEFAULT_CONTENT_TYPE; \
        if [[ -z "" ]]; then \
            ./node_modules/.bin/nodeunit test/*.test.js; \
        else \
            echo "# Running subset of tests matching TEST_FILTER="; \
            ./node_modules/.bin/nodeunit  test/client.test.js test/completion.test.js test/mfind.test.js test/mget.test.js test/mjob-simple.test.js test/mmpu.test.js test/mput.test.js test/muntar.test.js test/trackmarker.test.js test/utils.test.js; \
        fi

client.test.js
✔ mkdir
✔ mkdir (sub)
✔ put
✔ #231: put (special characters)
✔ #231: ls (special characters)
✔ #231: get (special characters)
✔ #231: rm (special characters)
✔ chattr
✔ put (zero byte streaming)
✔ put without mkdirp
✔ put with mkdirp
✔ streams
✔ put MD5 mismatch
✔ GH-72 content-length: undefined
✔ ls
✔ createListStream
✔ createListStream (dir only)
✔ createListStream (object only)
✔ ln
✔ info (link)
✔ ftw
✔ create job (simple grep)
✔ get job
✔ add input keys
✔ get job input
✔ end job
✔ wait for job
✔ get job output
✔ create and cancel job
✔ unlink object
✔ unlink link
✔ rmr
✔ mkdirp/rmr
✖ create upload

ResourceNotFoundError: /jhendricks/uploads does not exist
    at parseResponse (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/restify-clients/lib/JsonClient.js:73:26)
    at IncomingMessage.done (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/restify-clients/lib/StringClient.js:207:13)
    at IncomingMessage.g (events.js:199:16)
    at IncomingMessage.emit (events.js:129:20)
    at _stream_readable.js:908:16
    at process._tickCallback (node.js:355:11)

✖ upload part

AssertionError: 'undefined' === 'string'
    at MantaClient.uploadPart (/Users/jordan.hendricks/work/joyent-repos/node-manta/lib/client.js:3396:12)
    at Object.<anonymous> (/Users/jordan.hendricks/work/joyent-repos/node-manta/test/client.test.js:761:17)
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/lib/core.js:232:20
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:168:13
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:131:25
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:165:17
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:463:34
    at createClient (/Users/jordan.hendricks/work/joyent-repos/node-manta/test/client.test.js:74:9)
    at Object.module.exports.setUp (/Users/jordan.hendricks/work/joyent-repos/node-manta/test/client.test.js:78:9)
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/lib/core.js:260:35
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:458:21
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:163:13
    at iterate (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:123:13)
    at async.forEachSeries (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:139:9)
    at _asyncMap (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:162:9)
    at Object.mapSeries (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:152:23)

✖ get upload

AssertionError: 'undefined' === 'string'
    at MantaClient.getUpload (/Users/jordan.hendricks/work/joyent-repos/node-manta/lib/client.js:3516:12)
    at Object.<anonymous> (/Users/jordan.hendricks/work/joyent-repos/node-manta/test/client.test.js:785:17)
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/lib/core.js:232:20
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:168:13
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:131:25
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:165:17
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:463:34
    at createClient (/Users/jordan.hendricks/work/joyent-repos/node-manta/test/client.test.js:74:9)
    at Object.module.exports.setUp (/Users/jordan.hendricks/work/joyent-repos/node-manta/test/client.test.js:78:9)
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/lib/core.js:260:35
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:458:21
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:163:13
    at iterate (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:123:13)
    at async.forEachSeries (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:139:9)
    at _asyncMap (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:162:9)
    at Object.mapSeries (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:152:23)

✖ commit upload

AssertionError: 'undefined' === 'string'
    at MantaClient.commitUpload (/Users/jordan.hendricks/work/joyent-repos/node-manta/lib/client.js:3567:12)
    at Object.<anonymous> (/Users/jordan.hendricks/work/joyent-repos/node-manta/test/client.test.js:805:17)
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/lib/core.js:232:20
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:168:13
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:131:25
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:165:17
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:463:34
    at createClient (/Users/jordan.hendricks/work/joyent-repos/node-manta/test/client.test.js:74:9)
    at Object.module.exports.setUp (/Users/jordan.hendricks/work/joyent-repos/node-manta/test/client.test.js:78:9)
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/lib/core.js:260:35
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:458:21
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:163:13
    at iterate (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:123:13)
    at async.forEachSeries (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:139:9)
    at _asyncMap (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:162:9)
    at Object.mapSeries (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:152:23)

✖ abort upload

ResourceNotFoundError: /jhendricks/uploads does not exist
    at parseResponse (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/restify-clients/lib/JsonClient.js:73:26)
    at IncomingMessage.done (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/restify-clients/lib/StringClient.js:207:13)
    at IncomingMessage.g (events.js:199:16)
    at IncomingMessage.emit (events.js:129:20)
    at _stream_readable.js:908:16
    at process._tickCallback (node.js:355:11)

✖ #311: create upload with special headers

ResourceNotFoundError: /jhendricks/uploads does not exist
    at parseResponse (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/restify-clients/lib/JsonClient.js:73:26)
    at IncomingMessage.done (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/restify-clients/lib/StringClient.js:207:13)
    at IncomingMessage.g (events.js:199:16)
    at IncomingMessage.emit (events.js:129:20)
    at _stream_readable.js:908:16
    at process._tickCallback (node.js:355:11)

✔ GH-196 getPath ~~/
✔ #180: Invalid key results in no client error
✔ MANTA-2812 null signer
✔ MANTA-2812 undefined signer

completion.test.js
✔ mchattr --completion
✔ mchmod --completion
✔ mfind --completion
✔ mget --completion
✔ minfo --completion
✔ mjob --completion
✔ mln --completion
✔ mlogin --completion
✔ mls --completion
✔ mmd5 --completion
✔ mmkdir --completion
✔ mmpu --completion
✔ mput --completion
✔ mrm --completion
✔ mrmdir --completion
✔ msign --completion
✔ muntar --completion

mfind.test.js
✔ setup: create test tree at /jhendricks/stor/node-manta-test-mfind-ae765d98
✔ check if operator (mfind forbidden)
✔ mfind TESTDIR
✔ mfind -j TESTDIR
✔ mfind TESTDIR/afile.txt
✔ mfind TESTDIR/notafile.txt
✔ mfind TESTDIR/notafile.txt TESTDIR/afile.txt
WARNING: skipping mfind forbidden test: user is an operator
✔ mfind /poseidon/stor TESTDIR/afile.txt
✔ cleanup: rm test tree /jhendricks/stor/node-manta-test-mfind-ae765d98

mget.test.js
✔ setup: create test tree at /jhendricks/stor/node-manta-test-mget-380d867d
✔ mget TESTDIR/02.data
✔ mget TESTDIR/01.data TESTDIR/02.data TESTDIR/03.data
✔ mget TESTDIR/01.txt TESTDIR/02.txt TESTDIR/03.txt
✔ mget -o TMPFILE TESTDIR/01.txt TESTDIR/02.txt TESTDIR/03.txt
✔ mget TESTDIR/01.txt TESTDIR/02.txt TESTDIR/XX.txt TESTDIR/03.txt
✔ mget -o TMPFILE TESTDIR/01.txt TESTDIR/02.txt TESTDIR/XX.txt TESTDIR/03.txt
✔ cleanup: rm test tree /jhendricks/stor/node-manta-test-mget-380d867d

mjob-simple.test.js
✔ mjob create --close -or "echo hello"

mmpu.test.js
✖ mmpu create C_OBJ_PATH -c 1 -H m-custom-header:foo -s TEXT_SIZE -m TEXT_MD5

VError: exec "/Users/jordan.hendricks/work/joyent-repos/node-manta/bin/mmpu create /jhendricks/stor/node-manta-test-mmpu-jhendricks-commit -c 1 -H m-custom-header:foo -s 9 -m 3pNu6QUfS3RORVBpMh20lA==": exited with status 1
    at interpretChildProcessResult (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/forkexec/lib/forkexec.js:163:15)
    at onChildExited (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/forkexec/lib/forkexec.js:75:9)
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/forkexec/lib/forkexec.js:53:4
    at ChildProcess.exithandler (child_process.js:761:5)
    at ChildProcess.emit (events.js:110:17)
    at maybeClose (child_process.js:1019:16)
    at Socket.<anonymous> (child_process.js:1190:11)
    at Socket.emit (events.js:107:17)
    at Pipe.close (net.js:484:11)

✖ mmpu get C_ID

AssertionError: 'object' === '[string]'
    at forkExecWait (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/forkexec/lib/forkexec.js:22:17)
    at Object.<anonymous> (/Users/jordan.hendricks/work/joyent-repos/node-manta/test/mmpu.test.js:110:5)
    at Object.<anonymous> (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/lib/core.js:236:16)
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/lib/core.js:236:16
    at Object.exports.runTest (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/lib/core.js:70:9)
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/lib/core.js:118:25
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:513:13
    at iterate (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:123:13)
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:134:25
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:515:17
    at Immediate._onImmediate (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/lib/types.js:146:17)
    at processImmediate [as _immediateCallback] (timers.js:367:17)

✖ mmpu create A_OBJ_PATH -c 1 -s TEXT_SIZE -m TEXT_MD5-H durability-level 3 -H content-length:10 -H content-md5:foo

VError: exec "/Users/jordan.hendricks/work/joyent-repos/node-manta/bin/mmpu create /jhendricks/stor/node-manta-test-mmpu-jhendricks-abort -c 1 -s 9 -m 3pNu6QUfS3RORVBpMh20lA== -H durability-level:3 -H content-length:10 -H content-md5:foo": exited with status 1
    at interpretChildProcessResult (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/forkexec/lib/forkexec.js:163:15)
    at onChildExited (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/forkexec/lib/forkexec.js:75:9)
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/forkexec/lib/forkexec.js:53:4
    at ChildProcess.exithandler (child_process.js:761:5)
    at ChildProcess.emit (events.js:110:17)
    at maybeClose (child_process.js:1019:16)
    at Socket.<anonymous> (child_process.js:1190:11)
    at Socket.emit (events.js:107:17)
    at Pipe.close (net.js:484:11)

✖ mmpu get A_IDJ

AssertionError: 'object' === '[string]'
    at forkExecWait (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/forkexec/lib/forkexec.js:22:17)
    at Object.<anonymous> (/Users/jordan.hendricks/work/joyent-repos/node-manta/test/mmpu.test.js:177:5)
    at Object.<anonymous> (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/lib/core.js:236:16)
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/lib/core.js:236:16
    at Object.exports.runTest (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/lib/core.js:70:9)
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/lib/core.js:118:25
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:513:13
    at iterate (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:123:13)
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:134:25
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:515:17
    at Immediate._onImmediate (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/lib/types.js:146:17)
    at processImmediate [as _immediateCallback] (timers.js:367:17)

✖ mmpu parts C_ID: pre-upload

AssertionError: 'object' === '[string]'
    at forkExecWait (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/forkexec/lib/forkexec.js:22:17)
    at Object.<anonymous> (/Users/jordan.hendricks/work/joyent-repos/node-manta/test/mmpu.test.js:216:5)
    at Object.<anonymous> (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/lib/core.js:236:16)
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/lib/core.js:236:16
    at Object.exports.runTest (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/lib/core.js:70:9)
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/lib/core.js:118:25
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:513:13
    at iterate (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:123:13)
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:134:25
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:515:17
    at Immediate._onImmediate (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/lib/types.js:146:17)
    at processImmediate [as _immediateCallback] (timers.js:367:17)

✖ mmpu parts A_ID: pre-upload

AssertionError: 'object' === '[string]'
    at forkExecWait (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/forkexec/lib/forkexec.js:22:17)
    at Object.<anonymous> (/Users/jordan.hendricks/work/joyent-repos/node-manta/test/mmpu.test.js:235:5)
    at Object.<anonymous> (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/lib/core.js:236:16)
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/lib/core.js:236:16
    at Object.exports.runTest (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/lib/core.js:70:9)
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/lib/core.js:118:25
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:513:13
    at iterate (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:123:13)
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:134:25
    at /Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/deps/async.js:515:17
    at Immediate._onImmediate (/Users/jordan.hendricks/work/joyent-repos/node-manta/node_modules/nodeunit/lib/types.js:146:17)
    at processImmediate [as _immediateCallback] (timers.js:367:17)

FAILURES: Undone tests (or their setups/teardowns): 
- mmpu list

To fix this, make sure all tests call test.done()
make: *** [test] Error 1
➜  node-manta git:(master)

The test suite can detect whether MPU is supported by checking for the existence of the top-level "uploads" directory. If that directory isn't there, MPU tests should be skipped and print a warning message for the user that these tests were not run.

jordanhendricks commented 7 years ago

After some discussion on jabber, we've decided it's best to do this feature detection in the client. See #320.

jordanhendricks commented 7 years ago

CR at https://cr.joyent.us/#/c/2417/.

Testing Notes I ran the node-manta test suite against a deployment with MPU not enabled, as well as one without MPU enabled.

Here is the output from the tests when MPU is not enabled:

$  make test
npm install
unset MANTA_DEFAULT_CONTENT_TYPE; \
        if [[ -z "" ]]; then \
            ./node_modules/.bin/nodeunit test/*.test.js; \
        else \
            echo "# Running subset of tests matching TEST_FILTER="; \
            ./node_modules/.bin/nodeunit  test/client.test.js test/completion.test.js test/mfind.test.js test/mget.test.js test/mjob-simple.test.js test/mmpu.test.js test/mput.test.js test/muntar.test.js test/trackmarker.test.js test/utils.test.js; \
        fi

client.test.js
✔ mkdir
✔ mkdir (sub)
✔ put
✔ #231: put (special characters)
✔ #231: ls (special characters)
✔ #231: get (special characters)
✔ #231: rm (special characters)
✔ chattr
✔ put (zero byte streaming)
✔ put without mkdirp
✔ put with mkdirp
✔ streams
✔ put MD5 mismatch
✔ GH-72 content-length: undefined
✔ ls
✔ createListStream
✔ createListStream (dir only)
✔ createListStream (object only)
✔ ln
✔ info (link)
✔ ftw
✔ create job (simple grep)
✔ get job
✔ add input keys
✔ get job input
✔ end job
✔ wait for job
✔ get job output
✔ create and cancel job
✔ unlink object
✔ unlink link
✔ rmr
✔ mkdirp/rmr
WARNING: skipping test "create upload": multipart upload is not enabled on this Manta deployment
✔ create upload
WARNING: skipping test "upload part": multipart upload is not enabled on this Manta deployment
✔ upload part
WARNING: skipping test "get upload": multipart upload is not enabled on this Manta deployment
✔ get upload
WARNING: skipping test "commit upload": multipart upload is not enabled on this Manta deployment
✔ commit upload
WARNING: skipping test "abort upload": multipart upload is not enabled on this Manta deployment
✔ abort upload
WARNING: skipping test: "#311: create upload with special headers": multipart upload is not enabled on this Manta deployment
✔ #311: create upload with special headers
✔ GH-196 getPath ~~/
✔ #180: Invalid key results in no client error
✔ MANTA-2812 null signer
✔ MANTA-2812 undefined signer

completion.test.js
✔ mchattr --completion
✔ mchmod --completion
✔ mfind --completion
✔ mget --completion
✔ minfo --completion
✔ mjob --completion
✔ mln --completion
✔ mlogin --completion
✔ mls --completion
✔ mmd5 --completion
✔ mmkdir --completion
✔ mmpu --completion
✔ mput --completion
✔ mrm --completion
✔ mrmdir --completion
Connection to cr.joyent.us closed by remote host.
✔ msign --completion
✔ muntar --completion

mfind.test.js
✔ setup: create test tree at /jhendricks/stor/node-manta-test-mfind-d4c22156
✔ check if operator (mfind forbidden)
✔ mfind TESTDIR
✔ mfind -j TESTDIR
✔ mfind TESTDIR/afile.txt
✔ mfind TESTDIR/notafile.txt
✔ mfind TESTDIR/notafile.txt TESTDIR/afile.txt
WARNING: skipping mfind forbidden test: user is an operator
✔ mfind /poseidon/stor TESTDIR/afile.txt
✔ cleanup: rm test tree /jhendricks/stor/node-manta-test-mfind-d4c22156

mget.test.js
✔ setup: create test tree at /jhendricks/stor/node-manta-test-mget-92260d99
✔ mget TESTDIR/02.data
✔ mget TESTDIR/01.data TESTDIR/02.data TESTDIR/03.data
✔ mget TESTDIR/01.txt TESTDIR/02.txt TESTDIR/03.txt
✔ mget -o TMPFILE TESTDIR/01.txt TESTDIR/02.txt TESTDIR/03.txt
✔ mget TESTDIR/01.txt TESTDIR/02.txt TESTDIR/XX.txt TESTDIR/03.txt
✔ mget -o TMPFILE TESTDIR/01.txt TESTDIR/02.txt TESTDIR/XX.txt TESTDIR/03.txt
✔ cleanup: rm test tree /jhendricks/stor/node-manta-test-mget-92260d99

mjob-simple.test.js
✔ mjob create --close -or "echo hello"

mmpu.test.js
WARNING: skipping test: multipart upload is not enabled on this Manta deployment
✔ mmpu create C_OBJ_PATH -c 1 -H m-custom-header:foo -s TEXT_SIZE -m TEXT_MD5
WARNING: skipping test: multipart upload is not enabled on this Manta deployment
✔ mmpu get C_ID
WARNING: skipping test: multipart upload is not enabled on this Manta deployment
✔ mmpu create A_OBJ_PATH -c 1 -s TEXT_SIZE -m TEXT_MD5-H durability-level 3 -H content-length:10 -H content-md5:foo
WARNING: skipping test: multipart upload is not enabled on this Manta deployment
✔ mmpu get A_IDJ
WARNING: skipping test: multipart upload is not enabled on this Manta deployment
✔ mmpu parts C_ID: pre-upload
WARNING: skipping test: multipart upload is not enabled on this Manta deployment
✔ mmpu parts A_ID: pre-upload
WARNING: skipping test: multipart upload is not enabled on this Manta deployment
✔ mmpu list
WARNING: skipping test: multipart upload is not enabled on this Manta deployment
✔ mmpu upload C_ID 0 -f tmpFile
WARNING: skipping test: multipart upload is not enabled on this Manta deployment
✔ mmpu upload A_ID 0
WARNING: skipping test: multipart upload is not enabled on this Manta deployment
✔ mmpu parts C_ID: post-upload
WARNING: skipping test: multipart upload is not enabled on this Manta deployment
✔ mmpu parts A_ID: post-upload
WARNING: skipping test: multipart upload is not enabled on this Manta deployment
✔ mmpu list -p
WARNING: skipping test: multipart upload is not enabled on this Manta deployment
✔ mmpu list: post part upload
WARNING: skipping test: multipart upload is not enabled on this Manta deployment
✔ mmpu commit C_ID C_ETAG0
WARNING: skipping test: multipart upload is not enabled on this Manta deployment
✔ mmpu abort A_ID

mput.test.js
✔ setup: create test tree at /jhendricks/stor/node-manta-test-mput-fc517da8
✔ mput with custom header value with colons
✔ cleanup: rm test tree /jhendricks/stor/node-manta-test-mput-fc517da8
✔ cleanup: rm tmp directory /var/folders/c7/c5brblnd3l93659s9gkbxfbm0000gp/T/

muntar.test.js
✔ muntar case 0: corpus/tar1.tar

trackmarker.test.js
✔ pages by name
✔ pages by mtime

utils.test.js
✔ escapePath encoding

OK: 315 assertions (34832ms)

Here is the output from the tests when MPU is enabled:

$ make test
npm install
unset MANTA_DEFAULT_CONTENT_TYPE; \
        if [[ -z "" ]]; then \
            ./node_modules/.bin/nodeunit test/*.test.js; \
        else \
            echo "# Running subset of tests matching TEST_FILTER="; \
            ./node_modules/.bin/nodeunit  test/client.test.js test/completion.test.js test/mfind.test.js test/mget.test.js test/mjob-simple.test.js test/mmpu.test.js test/mput.test.js test/muntar.test.js test/trackmarker.test.js test/utils.test.js; \
        fi

client.test.js
✔ mkdir
✔ mkdir (sub)
✔ put
✔ #231: put (special characters)
✔ #231: ls (special characters)
✔ #231: get (special characters)
✔ #231: rm (special characters)
✔ chattr
✔ put (zero byte streaming)
✔ put without mkdirp
✔ put with mkdirp
✔ streams
✔ put MD5 mismatch
✔ GH-72 content-length: undefined
✔ ls
✔ createListStream
✔ createListStream (dir only)
✔ createListStream (object only)
✔ ln
✔ info (link)
✔ ftw
✔ create job (simple grep)
✔ get job
✔ add input keys
✔ get job input
✔ end job
✔ wait for job
✔ get job output
✔ create and cancel job
✔ unlink object
✔ unlink link
✔ rmr
✔ mkdirp/rmr
✔ create upload
✔ upload part
✔ get upload
✔ commit upload
✔ abort upload
✔ #311: create upload with special headers
✔ GH-196 getPath ~~/
✔ #180: Invalid key results in no client error
✔ MANTA-2812 null signer
✔ MANTA-2812 undefined signer

completion.test.js
✔ mchattr --completion
✔ mchmod --completion
✔ mfind --completion
✔ mget --completion
✔ minfo --completion
✔ mjob --completion
✔ mln --completion
✔ mlogin --completion
✔ mls --completion
✔ mmd5 --completion
✔ mmkdir --completion
✔ mmpu --completion
✔ mput --completion
✔ mrm --completion
✔ mrmdir --completion
✔ msign --completion
✔ muntar --completion

mfind.test.js
✔ setup: create test tree at /jhendricks/stor/node-manta-test-mfind-e37aa9c8
✔ check if operator (mfind forbidden)
✔ mfind TESTDIR
✔ mfind -j TESTDIR
✔ mfind TESTDIR/afile.txt
✔ mfind TESTDIR/notafile.txt
✔ mfind TESTDIR/notafile.txt TESTDIR/afile.txt
WARNING: skipping mfind forbidden test: user is an operator
✔ mfind /poseidon/stor TESTDIR/afile.txt
✔ cleanup: rm test tree /jhendricks/stor/node-manta-test-mfind-e37aa9c8

mget.test.js
✔ setup: create test tree at /jhendricks/stor/node-manta-test-mget-1859dcb3
✔ mget TESTDIR/02.data
✔ mget TESTDIR/01.data TESTDIR/02.data TESTDIR/03.data
✔ mget TESTDIR/01.txt TESTDIR/02.txt TESTDIR/03.txt
✔ mget -o TMPFILE TESTDIR/01.txt TESTDIR/02.txt TESTDIR/03.txt
✔ mget TESTDIR/01.txt TESTDIR/02.txt TESTDIR/XX.txt TESTDIR/03.txt
✔ mget -o TMPFILE TESTDIR/01.txt TESTDIR/02.txt TESTDIR/XX.txt TESTDIR/03.txt
✔ cleanup: rm test tree /jhendricks/stor/node-manta-test-mget-1859dcb3

mjob-simple.test.js
✔ mjob create --close -or "echo hello"

mmpu.test.js
✔ mmpu create C_OBJ_PATH -c 1 -H m-custom-header:foo -s TEXT_SIZE -m TEXT_MD5
✔ mmpu get C_ID
✔ mmpu create A_OBJ_PATH -c 1 -s TEXT_SIZE -m TEXT_MD5-H durability-level 3 -H content-length:10 -H content-md5:foo
✔ mmpu get A_IDJ
✔ mmpu parts C_ID: pre-upload
✔ mmpu parts A_ID: pre-upload
✔ mmpu list
✔ mmpu upload C_ID 0 -f tmpFile
✔ mmpu upload A_ID 0
✔ mmpu parts C_ID: post-upload
✔ mmpu parts A_ID: post-upload
✔ mmpu list -p
✔ mmpu list: post part upload
✔ mmpu commit C_ID C_ETAG0
✔ mmpu abort A_ID

mput.test.js
✔ setup: create test tree at /jhendricks/stor/node-manta-test-mput-e7e94c81
✔ mput with custom header value with colons
✔ cleanup: rm test tree /jhendricks/stor/node-manta-test-mput-e7e94c81
✔ cleanup: rm tmp directory /var/folders/c7/c5brblnd3l93659s9gkbxfbm0000gp/T/

muntar.test.js
✔ muntar case 0: corpus/tar1.tar

trackmarker.test.js
✔ pages by name
✔ pages by mtime

utils.test.js
✔ escapePath encoding

OK: 404 assertions (44608ms)
jordanhendricks commented 7 years ago

A new MPU test was added in a commit integrated before this change. As such, I have rebased this change and updated the MPU test (reflected in patchset 9 on the Gerrit CR).

I have re-run the test suite against a Manta both with and without MPU enabled. The test run against an MPU-enabled deployment passed. The test suite against a Manta without MPU passed except for the MPU tests, which skipped successfully.

Here is the MPU-related portion of the output from the test run against a non-MPU enabled Manta (note the addition of the "errant commit upload returns undefined res" test to the client tests):

...
client.test.js
✔ mkdir
✔ mkdir (sub)
✔ put
✔ #231: put (special characters)
✔ #231: ls (special characters)
✔ #231: get (special characters)
✔ #231: rm (special characters)
✔ chattr
✔ put (zero byte streaming)
✔ put without mkdirp
✔ put with mkdirp
✔ streams
✔ put MD5 mismatch
✔ GH-72 content-length: undefined
✔ ls
✔ createListStream
✔ createListStream (dir only)
✔ createListStream (object only)
✔ ln
✔ info (link)
✔ ftw
✔ create job (simple grep)
✔ get job
✔ add input keys
✔ get job input
✔ end job
✔ wait for job
✔ get job output
✔ create and cancel job
✔ unlink object
✔ unlink link
✔ rmr
✔ mkdirp/rmr
WARNING: skipping test "create upload": multipart upload is not enabled on this Manta deployment
✔ create upload
WARNING: skipping test "upload part": multipart upload is not enabled on this Manta deployment
✔ upload part
WARNING: skipping test "get upload": multipart upload is not enabled on this Manta deployment
✔ get upload
WARNING: skipping test "commit upload": multipart upload is not enabled on this Manta deployment
✔ commit upload
WARNING: skipping test "commit upload": multipart upload is not enabled on this Manta deployment
✔ errant commit upload returns undefined res
WARNING: skipping test "abort upload": multipart upload is not enabled on this Manta deployment
✔ abort upload
WARNING: skipping test: "#311: create upload with special headers": multipart upload is not enabled on this Manta deployment
✔ #311: create upload with special headers
✔ GH-196 getPath ~~/
✔ #180: Invalid key results in no client error
✔ MANTA-2812 null signer
✔ MANTA-2812 undefined signer
...
mmpu.test.js
WARNING: skipping test: multipart upload is not enabled on this Manta deployment
✔ mmpu create C_OBJ_PATH -c 1 -H m-custom-header:foo -s TEXT_SIZE -m TEXT_MD5
WARNING: skipping test: multipart upload is not enabled on this Manta deployment
✔ mmpu get C_ID
WARNING: skipping test: multipart upload is not enabled on this Manta deployment
✔ mmpu create A_OBJ_PATH -c 1 -s TEXT_SIZE -m TEXT_MD5-H durability-level 3 -H content-length:10 -H content-md5:foo
WARNING: skipping test: multipart upload is not enabled on this Manta deployment
✔ mmpu get A_IDJ
WARNING: skipping test: multipart upload is not enabled on this Manta deployment
✔ mmpu parts C_ID: pre-upload
WARNING: skipping test: multipart upload is not enabled on this Manta deployment
✔ mmpu parts A_ID: pre-upload
WARNING: skipping test: multipart upload is not enabled on this Manta deployment
✔ mmpu list
WARNING: skipping test: multipart upload is not enabled on this Manta deployment
✔ mmpu upload C_ID 0 -f tmpFile
WARNING: skipping test: multipart upload is not enabled on this Manta deployment
✔ mmpu upload A_ID 0
WARNING: skipping test: multipart upload is not enabled on this Manta deployment
✔ mmpu parts C_ID: post-upload
WARNING: skipping test: multipart upload is not enabled on this Manta deployment
✔ mmpu parts A_ID: post-upload
WARNING: skipping test: multipart upload is not enabled on this Manta deployment
✔ mmpu list -p
WARNING: skipping test: multipart upload is not enabled on this Manta deployment
✔ mmpu list: post part upload
WARNING: skipping test: multipart upload is not enabled on this Manta deployment
✔ mmpu commit C_ID C_ETAG0
WARNING: skipping test: multipart upload is not enabled on this Manta deployment
✔ mmpu abort A_ID
...