Find library or implement canonical json serializer

[cajun-rat]

If there isn't a library, we might need to extend one of the JSON libraries we've got in third_party to write canonical json.

[cajun-rat]

https://pypi.python.org/pypi/canonicaljson is the code that we are using as a reference

[patriotyk]

I have researched little bit and it looks like there is no such libraries for c++. But do we really need such library in client side? Do we need to generate new json? Or just use returned from server?

[taheris]

Yes, we need to generate a manifest file to send to the director.

[heartsucker]

Also, there is no way for the client to correctly verify signed metadata without a cjson lib either. It would be unreliable to assume that the server will send it in the right format.

[cajun-rat]

I'm actually a bit concerned about the security implications of the server sending anything that isn't in canonical form. I've seen bugs before where the bad guys were able to exploit differences in behaviour between parsers/normalisers to sneak changes past signature validation.

In the short term we might need this, but since the server has to be able to normalise anyway, it would be much cleaner if it just sent exactly the bytes that it signed. That has the nice advantage that the client can do a signature check before giving the (untrusted) input to a JSON parser.

We do still need the canonicaliser for sending the manifest to the server though.

[heartsucker]

Because of how the cjson is signed it would require someone to find a bug in the parser and exploit RSA/ed25519 (the two signing schemes we use) in order for this to break. Since the latter two are deemed safe, it shouldn't matter what form we initially send the json in.

Furthermore, the signature for the json is inside the json itself, so a full parse needs to happen to extract the data about the signature (keyid, method, sig) before we can even validate the signed portion of the json.

The risk here is assuming the servers always send cjson. We could DoS our devices by changing the backend. Or something like mod_pagespeed or some other proxy in between could reorder the json for reasons of compression (or anything else) and then we'd be up a creek.

[cajun-rat]

I just had a chat with @heartsucker about this, and we don't think we can make the server send canonical JSON. This means the client has to be very careful about differences between the behaviour of JSON parsers. Specifically, we need to:

1. Parse the incoming JSON from the server and extract the signature information and signed JSON object.
2. Serialise this object to a list of bytes and perform the signature check
3. Re-parse exactly those bytes into JSON, and then go and inspect the data from that

Note that we must not use the parse result in #1 inside the application: we should consider that to be 'untrusted'

[heartsucker]

@cajun-rat It's not that we can't make the server do it, it's that I think it would be too error prone in that a dev might easily just toss out plain json. Or, if we want this to be maximally compatible with other services, we shouldn't have the client reject all incoming json that isn't already cjson.

[patriotyk]

I have found the solution. Our jsoncpp has serializer almost compatible with cjson, only small fix is required. So I am preparing it.

[cajun-rat]

Nice. I'm happy for you to upstream this change to jsoncpp, if you wish