RFC: Custom Packet differenciation (aka packet ids) for NGC

[Green-Sky]

Toxcore made the choice of not requiring a packet id system in the api for custom packets. https://github.com/TokTok/c-toxcore/blob/9592d590cf4b4bad9117d9df618c7bab351cb70f/toxcore/tox.h#L4744-L4771

As such, here are a couple of possibilities, ideally one of which we choose to be THE ONLY way. There are however implicit compatibilities between different options.

static packet ids

integer packet id

A0 - 16bit fixed integer

Every packet is prefixed with a 16bit integer packet id. This allows for a total of 65536 possibilities. This also allows specify subranges for individual use and tox-offical "spec", like eg the first 1024 ids or more.

A1 and A2 - 24bit and 32bit fixed integer

Like option A0, but larger number space that is very unlikely to ever be exhausted/needed.

dual integer packet id

B0 - 8/16bit switched integer

Here we let the first bit decide whether the id is 8bit or 16bit (effectively 7bit and 15bit ranges). This allows for us to save an extra byte for packet types we deem profiting from this. eg. filetransfers will send a lot of data and ack type packets that can collectively waste bytes in the gigabytes ranges or more each year. Since the first bit is the switch, this only allows for 128 different ids, so they would be rare. It also cuts the larger id into 32768 possibilities. Which is probably still sufficient. Individually the ranges act similar to option A0 (and A1 and A2).

B1 and B2 - 8/24bit and 8/32bit switched integer

Like option B0, but larger second range. (also see option A1 and A2)

large integer packet id with namespace subdivision(s)

C0 - 64bit (48bit namespace + 8bit packet id + 8bit packet version)

This is conceptually like options A0, A1 and A2, but assignes different bytes different meanings. The first 6 bytes are used as a user/org specific namespace (aka magic), the next byte specifies the packet id and the last byte a packet version, for future compatibility checks. The advantage here is the clear separation between individuals and protocol changes. The disadvantage is the size, 64bit just to identify a packet type seems excessive.

C1 - 32bit (16bit namespace + 8bit packet id + 8bit packet version)

Same as option C0 but with a more reasonable namespace size. Still pretty large.

C2 - 24bit (12bit namespace + 12bit packet id)

Same as option C0 but with an even smaller namespace (4096) and smaller and combined packet id + packet version space (4096). The packet id and version amalgamation also means less space is wasted for the unlikely case of version filling the full 256 possibilities.

C3 - 8/24bit and 8/32bit switched namespaced integer

This is a combination of options B0 (B1, B2) and C2/C1. Where the 8bit(7bit) numbers are tox-official small packet id designated and the larger variant is option C2 or C1 with some bits moved around (either namespace or packet id(/version) has 1bit less).

dynamic packet ids

initial mapping announce

D0 - full packet id mapping exchanged at/after connection / before use

Here we exchange all the custom packets we understand and which ephemeral packet id to prefix each packet with. This might take multiple mapping announce packets. Then the other side knows which ephemeral id they should use to make data be treated as a certain packet type. Each packet type will be specified by a 64bit UUID (parts of which is a namespace). If a mapping announce packet is not ACKed, we resend the packet up to 3 times after each timeout. The mapping of a UUID to a ephemeral id is static, once announced. Later mapping announces can still add new mappings.

TODO: more variations of D0 ?

E0 - toxext (modified)

see https://github.com/toxext/toxext/blob/master/DESIGN.md With toxext, one sides sends UUID + MY_IDENTIFIER and the other side replies with YOUR_IDENTIFIER. The toxext magic will not be necessary, since we will designate every other use as off-spec.

---

If you have ideas for more variants, please let me know so I can add them to the list :mailbox_with_mail: . At some point we will have to discuss it more and settle on one of the options :speaking_head: . Timeline for this is to be settled on one of the options by the time v0.2.19 released (so very soon).

[zoff99]

dont forget to establish a package registry ala https://github.com/zoff99/toxcore_custom_packets_registry

[zoff99]

also i am using packages with these magic first 6 bytes already: 0x667788113435 so mark those as "already registered" please. depending on how many bytes it will be: 0x6677 if it will be the first 2

[Green-Sky]

also i am using packages with these magic first 6 bytes already: 0x667788113435 so mark those as "already registered" please. depending on how many bytes it will be: 0x6677 if it will be the first 2

yep, was my plan.

dont forget to establish a package registry ala https://github.com/zoff99/toxcore_custom_packets_registry

will do once we settle on the packet id type.

[Green-Sky]

Ok, ngc got a release. Pinging so we can move this along. I'd like to hear the opinion from @nurupo @JFreegman @iphydf @robinlinden and everyone interested :)