Risk Update Chunking - Githubissues

rlnsy commented 3 years ago

The update endpoint needs to divide the ephemeral id list into smaller chunks for downloading.

Specifically, we must do the following:

select a small-ish subset of ephemeral IDs and construct a cuckoo-filter that is no more that 2kb in size (this is a hard-cap for memory reasons)
generate a unique 'chunk number' (4 bytes) that corresponds to this filter
when an update is requested through the API, we write the chunk number, followed by the filter size (also 4 bytes), followed by the filter itself. The returned response will be at most 2048 + 4 + 4 = 2056 bytes.
this same filter and chunk number and length should be returned for a fixed amount of time (configurable parameter, on the order of seconds most likely), after which we cycle to a different chunk by repeating the steps above. - NOTE: the goal should be to cycle over all eph. IDs in the database (or at least the most recent ones)

rlnsy commented 3 years ago

EDIT: should allow chunk number as a parameter to the update request, which queries a specific chunk

rlnsy commented 3 years ago

EDIT: should also parameterize the max filter size e.g. MAX_FILTER_SIZE, but it should be set to 2kb right now as that's what the dongle assumes.

ubc-systopia / pancast-server