Decimal numbers don't convert to canonical JSON in the same way as Firefox's Canonical JSON

[mythmon]

To verify recipes executed on the client, we include a signature of the canonical JSON serialization of the data. The client can verify the signature by serializing it into JSON and verifying the signature against that. This relies on the server and client using the same serialization steps. This generally works fine.

It doesn't work fine for very small decimals, such as 0.00001. The Python implementation serialized this value as 1e-5, and Firefox's as '0.00001`. The server is much easier to change in this regard, so we should change it there.

As a side note, I don't think that canonical JSON is standardized anywhere, so we can't defer to authority on this.

CC @jvehent this is probably interesting to you.

[mostlygeek]

Are we using autograph for the signature?

The client can verify the signature by serializing it into JSON and verifying the signature against that.

Am I understanding this part correctly?:

• the "it" refers to a recipe js object after it has been deserialized
• this bug comes from the recipe-client re-serializing the recipe into JSON. Different JSON serializers result in different output
• the signatures won't match cause the bytes are different

The server is much easier to change in this regard, so we should change it there.

This sounds like the road to hell...

[jvehent]

Are we using autograph for the signature?

Yes, but autograph knows nothing of the input data. It just signs a bunch of bytes.

As a side note, I don't think that canonical JSON is standardized anywhere, so we can't defer to authority on this.

We talked about this a bunch when kinto implemented serialization, a couple years back, but never standardized it. Maybe it's time?

[mostlygeek]

Isn't RFC7159 the canonical JSON standard?

Are we asking if we should have a standard on the way JSON should be formatted so the bytes are always the same between different de/serializers?

[mythmon]

@mostlygeek, your understanding is correct. This is a difference between JSON serializers that are meant to be byte-for-byte identical.

RFC7159 is the canonical standard for JSON. What we're talking about here is "Canonical JSON", a subset of JSON that is intended to always produces the exact same bytes for a given object, which RFC7159 does not require.

For a simple example, {"a": 1, "b": 2} and {"b": 2, "a": 1} are equal under RFC7159, despite being different byte sequences. Canonical JSON would serialize both of these as {"a":1,"b":2} (keys sorted, no unneeded spaces).

The server is much easier to change in this regard, so we should change it there.

This sounds like the road to hell...

It's worked fine for us so far. More importantly, we aren't the only user of CanonicalJSON.jsm (the one in Firefox), and Normandy's serializer is used only by Normandy, and its only purpose is to match Firefox's. This is a pretty easy job, since Firefox's serializer hasn't changed in behavior since it was introduced in April 2016.

[mostlygeek]

What are the reasons for not checking the signature before we deserialize the JSON?

[jvehent]

Because the signature is stored inside the json.

Note that it could also be served as a http header, but we don't do that in Normandy or kinto.

On Mar 20, 2018 9:58 AM, "Benson Wong" notifications@github.com wrote:

What are the reasons for not checking the signature before we deserialize the JSON?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mozilla/normandy/issues/1210#issuecomment-374607523, or mute the thread https://github.com/notifications/unsubscribe-auth/AAZXgRXuW6hL3SSOqkMV1QQeORw8M-9Mks5tgQsfgaJpZM4Sw0Xg .

[mostlygeek]

Sending the signature so we don't have to de-re-serialize would be a lot better. Not having to use CanonicalJSON.jsm seems desirable.

[mythmon]

To expand on what @jvehent said, the payload we are verifying from the server looks like this (expanded to non-canonical JSON, to improve readability):

[
  {
    "recipe": { ... },  // this is the signed portion
    "signature": { ... }, // the signature for the recipe
  },
  { ... } // the recipe-signature pairs are repeated for each recipe
]

It would be technically possible to extract the raw bytes of the recipe key in each object from the response, but that would require a specialized JSON parser that doesn't exist in Firefox to my knowledge. I try and avoid writing parsers.

In order to serve the signature as a header, we would have to serve each recipe separately instead of in one request. This has negative effects on our servers (few big, consistent requests being easier than many small requests). We opted for the deserialize/serialize/verify step because it makes our servers scales better, and the code to support it on the client is robust enough.

I don't think that changing the signature verifying algorithm is in scope for this issue. If you'd still like to explore this change, please move this conversation to another issue or the mailing list.

[mythmon]

Closing this in favor of #1211.