Ability to return infected keys in binary format

[HendX]

Related to #48, the current method of returning keys in base 64 encoded JSON won't scale as well it should.

The API should be updated as follows:

• [ ] Ability to return application/octet-stream the infected keys. Either the server can decide to return this or the client can ask for it
• [ ] Update the app to handle binary data

Before jumping into this, some issues need to be addressed:

• [ ] How will since the since data be returned?
• [ ] How will the deleted_keys be returned?
• [ ] In #51, this could a new parameter in the JSON returned from infected endpoint. How would this be included in a binary stream?

I propose it be implemented something like the following:

• [ ] Include the since data in the response headers
• [ ] A header which includes:
  • [ ] The version number,
  • [ ] The number of keys / size of the key payload
  • [ ] The number of deleted keys / size of the key payload
  • [ ] The length of a key (16 bytes in the spec)
• [ ] Each key in the format: key data + UInt32. The specs indicate the key data is 16 bytes

[tatey]

Ability to return application/octet-stream the infected keys. Either the server can decide to return this or the client can ask for it

This is what the Accept header is for. The client can set multiple content types in order of most preferred to least preferred and the server can choose what to respond with.

$ curl -H "Accept: application/octet-stream, application/json" ...

the current method of returning keys in base 64 encoded JSON won't scale as well it should.

Would you consider going down the MsgPack route instead of rolling our own? The big advantage is that you could continue using the same data structure which you could encode and decode into JSON or encode and decode in MsgPack.

[HendX]

Yea that makes much more sense. I hadn’t heard of that, but yea something along those lines. I guess protocol buffers are somewhat similar.

I can definitely see that it needs to scale to handle large number of infected keys, but I’m hesitant to go too far in developing this just yet, since the features may be in some flux while the Apple API is still in development.

[HendX]

Glancing at the docs this looks far more straightforward than protobuf. Do you know if it’ll encode raw binary data so the keys don’t need to be base 64 encoded?

[tatey]

I believe so. Check out https://github.com/msgpack/msgpack/blob/master/spec.md#type-system in the spec.

[kevinrmblr]

Sounds like a good way to go. According to this issue, an accept header along the lines of application/msgpack could be used to use this over JSON.

Here's an active swift implementation, works directly on top of the Codable protocol, so it should be quite simply. As far as I know, raw Data should work fine, so instead of a (Base64) String, the DTO can use Data as key type?

[tatey]

Ahhh, that looks like a nice implementation. See this test https://github.com/Flight-School/MessagePack/blob/master/Tests/MessagePackTests/MessagePackDecodingTests.swift

[HendX]

Yep looks good. Just need to find a nice PHP implementation for my reference server. I'm going to work on this on a feature branch since it needs coordination between the server spec and the mobile app.

[HendX]

Ok I just got motivated and implemented MessagePack on both ends:

https://github.com/CrunchyBagel/TracePrivately/tree/feat/msgpack

I don't think I'm describing it correctly in the Swagger file though.

[HendX]

Since #53 is now merged I'm going to close this.