Support SHA256 key ids (and fix stale tests)

tgross commented 8 years ago

Newer versions of ssh-keygen write out SHA256 key fingerprints by default. The node.js client for Manta supports using the SHA256 fingerprint to identify the key but then canonicalizes the fingerprint to MD5 (ref https://github.com/joyent/node-smartdc-auth/pull/3). This PR uses the same method.

In order to verify this fix, I also had to fix two areas that were together causing all tests to fail:

The minimum index into the directory parts list for mkdirp had an off-by-one error, which means that the /:account/stor/tmp directory was being skipped over and never being created. This caused the entire test suite to fail.
The test suite assumed that the user has the man pages for mlogin and msign installed, but doesn't assert this is the case. They aren't installed as part of the base node-manta install and honestly I'm not even sure where to find them, so I've removed the assumption from the test of the help flags for mantash.

@trentm I didn't open a JIRA ticket for this because the library doesn't look like it's part of our core supported work based on the number of open issues. But I needed this fix sooner rather than later for the work I'm doing on the triton-mysql blueprint so rather than bothering you with this I just knocked it out on my own. Hope that's ok? If so, would you mind giving it a quick look to see if you're happy with it?

cc @misterbisson

tgross commented 8 years ago

Related: the documentation for using curl with Manta (ref https://apidocs.joyent.com/manta/api.html#interacting-with-the-joyent-manta-storage-service-from-the-shell-bash) doesn't point out the rather critical bit which is that it expects MANTA_KEY_ID to be an md5 hash. Where does that doc live so I can make an update to it?

trentm commented 8 years ago

Ah, thanks on the 'mkdir -p' off by one. For the record, here is the failure case before your patch:

$ mantash -v mkdir -p  /trent.mick/public/tmp-c/b/c/d/e
manta.client: DEBUG: PutDirectory '/trent.mick/public/tmp-c/b/c'
manta.auth: DEBUG: sign 'date: Tue, 29 Dec 2015 20:44:35 GMT' with agent key (algo rsa-sha1, fp b3:f0:a1:6c:18:3b:47:63:ae:6e:57:22:74:71:d4:bc)
manta.client: DEBUG: req: PUT /trent.mick/public/tmp-c/b/c
    host: us-east.manta.joyent.com
    headers: {'accept-encoding': 'gzip, deflate',
     'authorization': 'Signature keyId="/trent.mick/keys/b3:f0:a1:6c:18:3b:47:63:ae:6e:57:22:74:71:d4:bc",algorithm="rsa-sha1",signature="Yyq4E2HyaeLKQ0+pr+B81x6fIu4lJaIw1XqIibUvniXzBGHkzelJcr8ycCVtiYN2lXw17n5y/KCBW1IQ4OmLOri/L9xUINBNz6u35vTZor2Uf0UIHgfN8KcXx4zMvckK99mzA8y3iIQ2eCCOSzstabVtphMW+X3me4k6eOkUt5l+Bf1ORtUn7dwiodqmIiMjfGHCE9HZX6Mg5rei5OnxSYNazoQrjbfApcqiUhH+TGAqbqkV24R6AOfWBCJL+nBSE+4zbTV/fgL+Icu9F7TClfnU1IS3GA2dHanjNRurchrXreBwG6JXnQeKFjzZm05zRCHyDazkw97u6GWgdOz7Qw=="',
     'content-type': 'application/json; type=directory',
     'date': 'Tue, 29 Dec 2015 20:44:35 GMT',
     'user-agent': 'mantash/2.4.1 (darwin) Python/2.7.10'}
    body: (none)
manta.client: DEBUG: res: PUT /trent.mick/public/tmp-c/b/c
    {'connection': 'keep-alive',
     'content-length': '86',
     'content-md5': 'Y+tmxTlOl9M1e64GGGg+tw==',
     'content-type': 'application/json',
     'date': 'Tue, 29 Dec 2015 20:44:33 GMT',
     'server': 'Manta',
     'status': '404',
     'x-request-id': 'f769df80-ae6c-11e5-b335-79723e1d7e1d',
     'x-response-time': '11',
     'x-server-name': '60771e58-2ad0-4c50-8b23-86b72f9307f8'}
    {"code":"DirectoryDoesNotExist","message":"/trent.mick/public/tmp-c/b does not exist"}
manta.client: DEBUG: PutDirectory '/trent.mick/public/tmp-c/b'
manta.auth: DEBUG: sign 'date: Tue, 29 Dec 2015 20:44:36 GMT' with agent key (algo rsa-sha1, fp b3:f0:a1:6c:18:3b:47:63:ae:6e:57:22:74:71:d4:bc)
manta.client: DEBUG: req: PUT /trent.mick/public/tmp-c/b
    host: us-east.manta.joyent.com
    headers: {'accept-encoding': 'gzip, deflate',
     'authorization': 'Signature keyId="/trent.mick/keys/b3:f0:a1:6c:18:3b:47:63:ae:6e:57:22:74:71:d4:bc",algorithm="rsa-sha1",signature="ORuMW0glZszTx9Zwo1JaeDMzqsclWtPfLJ30p1EhTr1bIrr4CmbRHKsSQwv0Vk1GoJsw+BvKbldYv8JewPPKlpnsO+8WmP02UCyYbdAY/dcXTdWojtARpHxz6/OYgn+yalV6zKoVI1oOiYjT/LNyyQiheP+LAsI2shy5gexmAByt/oA8Wk5oY3BW0IRV89Waz617Tu37eKl3BCU6UGQa+MHPs6Wo+1w5ywjaH+pXc/xXvuaoUVviKOzT7t4Ug3awUJ+iGbPudXbalxKYuFuUpHYH33YOxza1r7ULcRVAQZrptd2TFO5u6YgHTlTMlwxLzNig/cBpHz1sB8+HSSI1kw=="',
     'content-type': 'application/json; type=directory',
     'date': 'Tue, 29 Dec 2015 20:44:36 GMT',
     'user-agent': 'mantash/2.4.1 (darwin) Python/2.7.10'}
    body: (none)
manta.client: DEBUG: res: PUT /trent.mick/public/tmp-c/b
    {'connection': 'keep-alive',
     'content-length': '84',
     'content-md5': 'wCBrT7c7VkHBxmqS9KQzIg==',
     'content-type': 'application/json',
     'date': 'Tue, 29 Dec 2015 20:44:33 GMT',
     'server': 'Manta',
     'status': '404',
     'x-request-id': 'f776b0c0-ae6c-11e5-b335-79723e1d7e1d',
     'x-response-time': '7',
     'x-server-name': '60771e58-2ad0-4c50-8b23-86b72f9307f8'}
    {"code":"DirectoryDoesNotExist","message":"/trent.mick/public/tmp-c does not exist"}
manta.client: DEBUG: PutDirectory '/trent.mick/public/tmp-c/b'
manta.auth: DEBUG: sign 'date: Tue, 29 Dec 2015 20:44:36 GMT' with agent key (algo rsa-sha1, fp b3:f0:a1:6c:18:3b:47:63:ae:6e:57:22:74:71:d4:bc)
manta.client: DEBUG: req: PUT /trent.mick/public/tmp-c/b
    host: us-east.manta.joyent.com
    headers: {'accept-encoding': 'gzip, deflate',
     'authorization': 'Signature keyId="/trent.mick/keys/b3:f0:a1:6c:18:3b:47:63:ae:6e:57:22:74:71:d4:bc",algorithm="rsa-sha1",signature="ORuMW0glZszTx9Zwo1JaeDMzqsclWtPfLJ30p1EhTr1bIrr4CmbRHKsSQwv0Vk1GoJsw+BvKbldYv8JewPPKlpnsO+8WmP02UCyYbdAY/dcXTdWojtARpHxz6/OYgn+yalV6zKoVI1oOiYjT/LNyyQiheP+LAsI2shy5gexmAByt/oA8Wk5oY3BW0IRV89Waz617Tu37eKl3BCU6UGQa+MHPs6Wo+1w5ywjaH+pXc/xXvuaoUVviKOzT7t4Ug3awUJ+iGbPudXbalxKYuFuUpHYH33YOxza1r7ULcRVAQZrptd2TFO5u6YgHTlTMlwxLzNig/cBpHz1sB8+HSSI1kw=="',
     'content-type': 'application/json; type=directory',
     'date': 'Tue, 29 Dec 2015 20:44:36 GMT',
     'user-agent': 'mantash/2.4.1 (darwin) Python/2.7.10'}
    body: (none)
manta.client: DEBUG: res: PUT /trent.mick/public/tmp-c/b
    {'connection': 'keep-alive',
     'content-length': '84',
     'content-md5': 'wCBrT7c7VkHBxmqS9KQzIg==',
     'content-type': 'application/json',
     'date': 'Tue, 29 Dec 2015 20:44:33 GMT',
     'server': 'Manta',
     'status': '404',
     'x-request-id': 'f784e190-ae6c-11e5-b335-79723e1d7e1d',
     'x-response-time': '7',
     'x-server-name': '60771e58-2ad0-4c50-8b23-86b72f9307f8'}
    {"code":"DirectoryDoesNotExist","message":"/trent.mick/public/tmp-c does not exist"}
mantash: ERROR: (DirectoryDoesNotExist) /trent.mick/public/tmp-c does not exist

trentm commented 8 years ago

published as version 2.4.1 to pypi. Thanks!

trentm commented 8 years ago

Where does that doc live so I can make an update to it?

An unfortunate story. Originally it was Markdown content from git.joyent.com:mantadoc.git, which itself pulled parts from node-manta.git and manta-muskie.git and rendered that as a single docset to HTML, which was published. However, the apidocs.jo was re-organized (new repo for new publishing system) but without yet a clean story for Manta docs import. Currently it is just a snapshot of the built mantadoc docs (HTML) and then with the odd post-edit directly to the HTML. :|

So you particular file would need to be edited here in both places:

tgross commented 8 years ago

@trentm I opened https://github.com/joyent/manta-muskie/pull/1 but it turns out I don't have access to https://github.com/joyent/apidocs.joyent.com

TritonDataCenter / python-manta

Support SHA256 key ids (and fix stale tests) #34