search

TritonDataCenter / node-manta

Node.js SDK for Manta
75 stars 54 forks source link

`mmpu commit` does not parse options #308

Closed jordanhendricks closed 7 years ago

jordanhendricks commented 7 years ago

The commit subcommand of mmpu does not parse options. Unlike the other subcommands (create, upload, abort, get, list, parts), it does not inherit the default node-manta options, which results in it having no options parsed at all.

Below are some examples of error messages one might get while trying to use mmpu commit with options. The examples show what should be equivalent invocations of a commit command, but we see different error messages with each. All examples are committing a single part for the same upload while also using the -a flag.

The reason we see differing error messages across invocations is that muskie validates part sizes and etags for all parts concurrently, and returns the first error encountered to the client. The common issue resulting in these errors is that mmpu is interpreting all arguments as the upload ID and list of etags, and sending those values to muskie in the commit.

In this example, muskie returns an error for part 1, which mmpu sent as -a.

➜  node-manta git:(master) ✗ mmpu commit $id a31c876e-7636-6576-a030-f2c8f0ef8e50 -a jhendricks
mmpu commit: error: jsonClient POST: upload fa105324-e4ee-eb41-91ce-f79e1951e5d7: [part 1] part does not exist

Here muskie notices that part 0, while the correct etag for that part, is too small to hit the minimum part size for an upload. This is because mmpu tried to commit two additional parts, with etags -a and jhendricks.

➜  node-manta git:(master) ✗ mmpu commit $id a31c876e-7636-6576-a030-f2c8f0ef8e50 -a jhendricks
mmpu commit: error: jsonClient POST: upload fa105324-e4ee-eb41-91ce-f79e1951e5d7: [part 0] part is too small (size 5278)

We can see a similar issue to the first example for other arguments interpreted as etags. Here muskie is trying to resolve the etag a31c876e-7636-6576-a030-f2c8f0ef8e50 as part 2, when it is in fact part 0.

➜  node-manta git:(master) ✗ mmpu commit -a jhendricks $id a31c876e-7636-6576-a030-f2c8f0ef8e50
mmpu commit: error: jsonClient POST: upload fa105324-e4ee-eb41-91ce-f79e1951e5d7: [part 2] part does not exist

If the flags come before the arguments, then we hit an assertion error in the client, which is interpreting -a as a the upload ID (and thus should be a uuid):

➜  node-manta git:(master) ✗ mmpu commit -a jhendricks $id a31c876e-7636-6576-a030-f2c8f0ef8e50
mmpu: error: id (uuid) is required
jordanhendricks commented 7 years ago

CR is at: https://cr.joyent.us/#/c/1997/.

Testing Notes: I ran the node-manta test suite. Additionally, I manually tried several combinations of flags, with my MANTA_USER and MANTA_URL unset.

No flags on mmpu commit (environment variables set):

# 0-part file
root@node-manta $ id=$(bin/mmpu create ~~/stor/308-test/no-args)
root@node-manta $ mmpu commit $id
root@node-manta $ mget ~~/stor/308-test/no-args 

# 1-part file
root@node-manta $ id=$(bin/mmpu create ~~/stor/308-test/one-part-no-flags)    
root@node-manta $ bin/mmpu upload $id 0 -f 5mb.out 
5mb.out                                                              [=======================================================================================================================================================================>] 100%   5.00MB                  
a57334d2-fb69-4395-ad84-763051589bae
root@node-manta $ bin/mmpu commit $id a57334d2-fb69-4395-ad84-763051589bae

Flags before the arguments:

# 0-part file
root@node-manta $ id=$(bin/mmpu create ~~/stor/308-test/zero-parts-flags-before  -u http://172.27.13.143:8080 -a jhendricks)
root@node-manta $ bin/mmpu commit -u http://172.27.13.143:8080 -a jhendricks $id

# 1-part file
root@node-manta $ id=$(bin/mmpu create ~~/stor/308-test/one-part-flags-before  -u http://172.27.13.143:8080 -a jhendricks)
root@node-manta $ bin/mmpu upload $id 0 -f 5mb.out  -u http://172.27.13.143:8080 -a jhendricks
5mb.out                                                              [=======================================================================================================================================================================>] 100%   5.00MB                  
02be07c5-d896-4fae-b683-a88931d1d74b
root@node-manta $ bin/mmpu commit -u http://172.27.13.143:8080 -a jhendricks $id 02be07c5-d896-4fae-b683-a88931d1d74b

# 2-part file
root@node-manta $ id=$(bin/mmpu create ~~/stor/308-test/two-parts-flags-before -u http://172.27.13.143:8080 -a jhendricks)
root@node-manta $ bin/mmpu upload $id 0 -f 5mb.out  -u http://172.27.13.143:8080 -a jhendricks
5mb.out                                                              [=======================================================================================================================================================================>] 100%   5.00MB                  
acba8a17-fe01-4e46-b781-0300e4b2a00c
root@node-manta $ bin/mmpu upload $id 1 -f 5mb.out  -u http://172.27.13.143:8080 -a jhendricks
5mb.out                                                              [=======================================================================================================================================================================>] 100%   5.00MB                  
6ac7ba74-7ac8-47d2-8057-3886eeaf6f50
root@node-manta $ bin/mmpu commit -u http://172.27.13.143:8080 -a jhendricks $id acba8a17-fe01-4e46-b781-0300e4b2a00c 6ac7ba74-7ac8-47d2-8057-3886eeaf6f50

Flags after the arguments:

# 0-part file
root@node-manta $ id=$(bin/mmpu create ~~/stor/308-test/zero-parts-flags-after  -u http://172.27.13.143:8080 -a jhendricks)
root@node-manta $ bin/mmpu commit $id -a jhendricks -u http://172.27.13.143:8080

# 1-part file
root@node-manta $ id=$(bin/mmpu create ~~/stor/308-test/one-part-flags-after  -u http://172.27.13.143:8080 -a jhendricks)
root@node-manta $ bin/mmpu commit $id 1d9e186d-5728-41f9-90a1-68dd80b3abf3 -a jhendricks -u http://172.27.13.143:8080

# 2-part file
root@node-manta $ id=$(bin/mmpu create ~~/stor/308-test/two-parts-flags-after -u http://172.27.13.143:8080 -a jhendricks)
root@node-manta $ bin/mmpu upload $id 0 -f 5mb.out -u http://172.27.13.143:8080 -a jhendricks
5mb.out                                                              [=======================================================================================================================================================================>] 100%   5.00MB                  
7da2bd09-cf48-4900-9497-464afe5f9bde
root@node-manta $ bin/mmpu upload $id 1 -f 5mb.out -u http://172.27.13.143:8080 -a jhendricks
5mb.out                                                              [=======================================================================================================================================================================>] 100%   5.00MB                  
b1558c26-94cf-4b87-9103-cbece965019a
root@node-manta $ bin/mmpu commit $id 7da2bd09-cf48-4900-9497-464afe5f9bde b1558c26-94cf-4b87-9103-cbece965019a -a jhendricks -u http://172.27.13.143:8080

A mixture of flags before and after arguments:

# 0-part file
root@node-manta $ id=$(bin/mmpu create ~~/stor/308-test/zero-parts-mixed-flags -u http://172.27.13.143:8080 -a jhendricks)
root@node-manta $ bin/mmpu commit -a jhendricks $id -u http://172.27.13.143:8080

# 1-part file
root@node-manta $ id=$(bin/mmpu create ~~/stor/308-test/one-part-mixed-flags -u http://172.27.13.143:8080 -a jhendricks)
root@node-manta $ bin/mmpu upload $id 0 -f 5mb.out -u http://172.27.13.143:8080 -a jhendricks
5mb.out                                                              [=======================================================================================================================================================================>] 100%   5.00MB                  
4d02332c-c3c6-41f4-87ed-3e5bf52817f5
root@node-manta $ bin/mmpu commit -a jhendricks $id 4d02332c-c3c6-41f4-87ed-3e5bf52817f5 -u http://172.27.13.143:8080

# 2-part file
root@node-manta $ id=$(bin/mmpu create ~~/stor/308-test/two-parts-mixed-flags -u http://172.27.13.143:8080 -a jhendricks)
root@node-manta $ bin/mmpu upload $id 0 -f 5mb.out -u http://172.27.13.143:8080 -a jhendricks
5mb.out                                                              [=======================================================================================================================================================================>] 100%   5.00MB                  
a71daacf-490c-4623-938d-5fabb59c7279
root@node-manta $ bin/mmpu upload $id 1 -f 5mb.out -u http://172.27.13.143:8080 -a jhendricks
5mb.out                                                              [=======================================================================================================================================================================>] 100%   5.00MB                  
9826863e-8bd1-47f2-b113-db8264499d5c
root@node-manta $ bin/mmpu commit -a jhendricks $id a71daacf-490c-4623-938d-5fabb59c7279 9826863e-8bd1-47f2-b113-db8264499d5c -u http://172.27.13.143:8080