search

notaryproject / notary

Notary is a project that allows anyone to have trust over arbitrary collections of data
Apache License 2.0
3.23k stars 511 forks source link

If I pass a file containing JSON to `notary add --custom`, it causes a verification error #1411

Open technosophos opened 5 years ago

technosophos commented 5 years ago

I was testing out the --custom feature when I ran into a problem that seems to stem from the fact that my custom data is JSON.

If I add simple data with --custom, things work fine. For example, the following target was added without a problem:

$ notary --tlscacert=$NOTARY_CA -s https://localhost:4443 -d ~/.duffle/tuf add hellohelm 0.1.4 bundle.json --custom simple.txt
Addition of target "0.1.4" to repository "hellohelm" staged for next publish.

This produces the JSON:

{
  "length": 751,
  "hashes": {
    "sha256": "aTubHxhVq6Q/4u8d/h3eQLRxThhX9rQ7BxOvCSyu2IM=",
    "sha512": "3xjlIieNrTYqJcdmo86GcnxoJ/ppW2Z7KgshXhRkwOeYCnwlIomJi4vmSKuwru708Uq6vyBmzWQsVr4MnkFb/g=="
  },
  "custom": 123
}

Note that custom is just 123 now.

$ notary --tlscacert=$NOTARY_CA -s https://localhost:4443 -d ~/.duffle/tuf publish hellohelm
Pushing changes to hellohelm
Enter passphrase for targets key with ID 1d7cc3e:
Enter passphrase for snapshot key with ID c4309d1:
Successfully published changes for repository hellohelm

I can see the custom data in the targets.json file:

{
  "signed": {
    "_type": "Targets",
    "delegations": {
      "keys": {},
      "roles": []
    },
    "expires": "2021-12-26T16:53:05.798405002-07:00",
    "targets": {
      "0.1.1": "/* removed this and other versions*/"
      "0.1.4": {
        "custom": 123,
        "hashes": {
          "sha256": "aTubHxhVq6Q/4u8d/h3eQLRxThhX9rQ7BxOvCSyu2IM=",
          "sha512": "3xjlIieNrTYqJcdmo86GcnxoJ/ppW2Z7KgshXhRkwOeYCnwlIomJi4vmSKuwru708Uq6vyBmzWQsVr4MnkFb/g=="
        },
        "length": 751
      }
    },
    "version": 5
  },
  "signatures": [
    {
      "keyid": "1d7cc3e53c63bad996ee56f0c6312409920ae358a6b0f63ffd0d64ea27f7140c",
      "method": "ecdsa",
      "sig": "gbly5OHEYaUJ6pc6Sz/v0hgf8UdAF5C1363oNGyDrjiKGHA2rLS/Qkgo1nz/uV+EtXOfiMeL++UzMbN3SyPI5A=="
    }
  ]
}

But if I try to add a file that contains JSON data, things fail in an interesting way:

$ notary --tlscacert=$NOTARY_CA -s https://localhost:4443 -d ~/.duffle/tuf add hellohelm 0.1.5 bundle.json --custom bundle.json
Addition of target "0.1.5" to repository "hellohelm" staged for next publish.

In the changelist, the data looks like this:

{"action":"create","role":"targets","type":"target","path":"0.1.5","data":"eyJsZW5ndGgiOjc1MSwiaGFzaGVzIjp7InNoYTI1NiI6ImFUdWJIeGhWcTZRLzR1OGQvaDNlUUxSeFRoaFg5clE3QnhPdkNTeXUySU09Iiwic2hhNTEyIjoiM3hqbElpZU5yVFlxSmNkbW84NkdjbnhvSi9wcFcyWjdLZ3NoWGhSa3dPZVlDbndsSW9tSmk0dm1TS3V3cnU3MDhVcTZ2eUJteldRc1ZyNE1ua0ZiL2c9PSJ9LCJjdXN0b20iOnsibmFtZSI6ImhlbGxvaGVsbSIsInZlcnNpb24iOiIwLjEuMCIsImludm9jYXRpb25JbWFnZXMiOlt7ImltYWdlVHlwZSI6ImRvY2tlciIsImltYWdlIjoiY25hYi9oZWxsb2hlbG06bGF0ZXN0In1dLCJpbWFnZXMiOlt7ImRlc2NyaXB0aW9uIjoiYWxwaW5lIiwiaW1hZ2UiOiJ0ZWNobm9zb3Bob3MvZGVtbzJhbHBpbmU6MC4xLjAiLCJpbWFnZVR5cGUiOiJkb2NrZXIiLCJyZWZzIjpbeyJwYXRoIjoiY25hYi9hcHAvY2hhcnRzL2FscGluZS92YWx1ZXMueWFtbCIsImZpZWxkIjoiaW1hZ2UucmVwb3NpdG9yeSJ9XX1dLCJwYXJhbWV0ZXJzIjp7InBvcnQiOnsiZGVmYXVsdFZhbHVlIjo4MDgwLCJ0eXBlIjoiaW50In19LCJjcmVkZW50aWFscyI6eyJrdWJlY29uZmlnIjp7InBhdGgiOiIvcm9vdC8ua3ViZS9jb25maWcifX19fQ=="}

For convenience, here's the decoded data, formatted:

{
  "length": 751,
  "hashes": {
    "sha256": "aTubHxhVq6Q/4u8d/h3eQLRxThhX9rQ7BxOvCSyu2IM=",
    "sha512": "3xjlIieNrTYqJcdmo86GcnxoJ/ppW2Z7KgshXhRkwOeYCnwlIomJi4vmSKuwru708Uq6vyBmzWQsVr4MnkFb/g=="
  },
  "custom": {
    "name": "hellohelm",
    "version": "0.1.0",
    "invocationImages": [
      {
        "imageType": "docker",
        "image": "cnab/hellohelm:latest"
      }
    ],
    "images": [
      {
        "description": "alpine",
        "image": "technosophos/demo2alpine:0.1.0",
        "imageType": "docker",
        "refs": [
          {
            "path": "cnab/app/charts/alpine/values.yaml",
            "field": "image.repository"
          }
        ]
      }
    ],
    "parameters": {
      "port": {
        "defaultValue": 8080,
        "type": "int"
      }
    },
    "credentials": {
      "kubeconfig": {
        "path": "/root/.kube/config"
      }
    }
  }
}

But now if I try to publish this, I get an error:

$ notary --tlscacert=$NOTARY_CA -s https://localhost:4443 -d ~/.duffle/tuf publish hellohelm
Pushing changes to hellohelm
Enter passphrase for targets key with ID 1d7cc3e:
Enter passphrase for snapshot key with ID c4309d1:

* fatal: The targets metadata is invalid: valid signatures did not meet threshold for targets

I see the following errors in the notary server log:

server_1  | {"level":"debug","msg":"entered ValidateRoot with dns: hellohelm","time":"2018-12-28T00:08:04Z"}
server_1  | {"level":"debug","msg":"found the following root keys: [7637a251a09fe1ae5236b7da7deb02973813b46aec18e400803edfa16354a200]","time":"2018-12-28T00:08:04Z"}
server_1  | {"level":"debug","msg":"found 1 valid leaf certificates for hellohelm: 7637a251a09fe1ae5236b7da7deb02973813b46aec18e400803edfa16354a200","time":"2018-12-28T00:08:04Z"}
server_1  | {"level":"debug","msg":"found 1 leaf certs, of which 1 are valid leaf certs for hellohelm","time":"2018-12-28T00:08:04Z"}
server_1  | {"level":"debug","msg":"checking root against trust_pinning config for hellohelm","time":"2018-12-28T00:08:04Z"}
server_1  | {"level":"debug","msg":"checking trust-pinning for cert: 7637a251a09fe1ae5236b7da7deb02973813b46aec18e400803edfa16354a200","time":"2018-12-28T00:08:04Z"}
server_1  | {"level":"debug","msg":" role has key IDs: 7637a251a09fe1ae5236b7da7deb02973813b46aec18e400803edfa16354a200","time":"2018-12-28T00:08:04Z"}
server_1  | {"level":"debug","msg":"verifying signature for key ID: 7637a251a09fe1ae5236b7da7deb02973813b46aec18e400803edfa16354a200","time":"2018-12-28T00:08:04Z"}
server_1  | {"level":"debug","msg":"root validation succeeded for hellohelm","time":"2018-12-28T00:08:04Z"}
server_1  | {"level":"debug","msg":"targets role has key IDs: 1d7cc3e53c63bad996ee56f0c6312409920ae358a6b0f63ffd0d64ea27f7140c","time":"2018-12-28T00:08:04Z"}
server_1  | {"level":"debug","msg":"verifying signature for key ID: 1d7cc3e53c63bad996ee56f0c6312409920ae358a6b0f63ffd0d64ea27f7140c","time":"2018-12-28T00:08:04Z"}
server_1  | {"level":"debug","msg":"failed ECDSA signature validation","time":"2018-12-28T00:08:04Z"}
server_1  | {"level":"debug","msg":"continuing b/c signature was invalid","time":"2018-12-28T00:08:04Z"}
server_1  | {"level":"error","msg":"ErrBadTargets: valid signatures did not meet threshold for targets","time":"2018-12-28T00:08:04Z"}
server_1  | {"go.version":"go1.10.3","http.request.contenttype":"multipart/form-data; boundary=2d7e6d0f9983aee76f1dc893f0c485185b17310d083e64b78d41c2692378","http.request.host":"localhost:4443","http.request.id":"cf0efc6a-6ad4-4ab0-8b6b-198c7e658ee4","http.request.method":"POST","http.request.remoteaddr":"172.18.0.1:44280","http.request.uri":"/v2/hellohelm/_trust/tuf/","http.request.useragent":"Go-http-client/1.1","level":"info","msg":"invalid update: Update sent by the client is invalid.: \u0026{ErrBadTargets The targets metadata is invalid: valid signatures did not meet threshold for targets}","time":"2018-12-28T00:08:04Z"}
server_1  | {"go.version":"go1.10.3","http.request.contenttype":"multipart/form-data; boundary=2d7e6d0f9983aee76f1dc893f0c485185b17310d083e64b78d41c2692378","http.request.host":"localhost:4443","http.request.id":"cf0efc6a-6ad4-4ab0-8b6b-198c7e658ee4","http.request.method":"POST","http.request.remoteaddr":"172.18.0.1:44280","http.request.uri":"/v2/hellohelm/_trust/tuf/","http.request.useragent":"Go-http-client/1.1","http.response.contenttype":"application/json; charset=utf-8","http.response.duration":"4.8472ms","http.response.status":400,"http.response.written":193,"level":"info","msg":"response completed","time":"2018-12-28T00:08:04Z"}

Version Info

Client version:

notary
 Version:    0.6.1
 Git commit: d6e1431f

Servers were built from master at 6e8cc6e54bb1fb140c97698993509aa5dfb1b9b8 using docker-compose build.

doanac commented 5 years ago

I'm hitting the same thing. I get even more odd behavior. Some JSON is accepted depending on some magic combination of JSON key names. So this works:

{
  "customType": "OSTREE",
  "theUrl": "https://github.com/foundriesio/lmp-manifest/releases/tag/37"
}

But this fails:

{
  "customType": "OSTREE",
  "theUrl": "https://github.com/foundriesio/lmp-manifest/releases/tag/37",
  "osTree": "https://api.foundries.io/lmp/treehub/release/api/v2/"
}

I'm starting to get the feeling it will only accept custom data with < 3 key names.

justincormack commented 5 years ago

Cc @riyazdf

On Tue, 8 Jan 2019 at 09:55, Andy Doan notifications@github.com wrote:

I'm hitting the same thing. I get even more odd behavior. Some JSON is accepted depending on some magic combination of JSON key names. So this works:

{ "customType": "OSTREE", "theUrl": "https://github.com/foundriesio/lmp-manifest/releases/tag/37" }

But this fails:

{ "customType": "OSTREE", "theUrl": "https://github.com/foundriesio/lmp-manifest/releases/tag/37", "osTree": "https://api.foundries.io/lmp/treehub/release/api/v2/" }

I'm starting to get the feeling it will only accept custom data with < 3 key names.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/theupdateframework/notary/issues/1411#issuecomment-452391989, or mute the thread https://github.com/notifications/unsubscribe-auth/AAdcPNXQO4OXh05md9P8yOgpz_kpSctdks5vBNubgaJpZM4ZjTuX .

technosophos commented 5 years ago

I'm wondering if there is some additional canonicalization that must happen to the JSON data in order for the hashing to match on repeat tests.

doanac commented 5 years ago

I can confirm @technosophos theory is correct. I've found that by changing the order of the keys, I can eventually get my custom data to publish.

justincormack commented 5 years ago

It may need to be Canonical JSON. Sorry at an offsite this week so haven’t had a chance to look at the code. There is a canonical json library Notary uses that you can use, or try doing it by hand...

On Wed, 9 Jan 2019 at 08:59, Matt Butcher notifications@github.com wrote:

I'm wondering if there is some additional canonicalization that must happen to the JSON data in order for the hashing to match on repeat tests.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/theupdateframework/notary/issues/1411#issuecomment-452760113, or mute the thread https://github.com/notifications/unsubscribe-auth/AAdcPPxSvIa22mPNZNLxcqbMK7JJchpXks5vBh_kgaJpZM4ZjTuX .

doanac commented 5 years ago

Thanks guys. That's my issue specifically quoting http://wiki.laptop.org/go/Canonical_JSON:

The members production in object must consist of keys in lexicographically sorted order