search

endlessm / azafea

Service to track device activations and usage metrics
Mozilla Public License 2.0
10 stars 2 forks source link

Avoid incrementing channel ID sequence if not needed #176

Closed wjt closed 3 months ago

wjt commented 1 year ago

When processing a submitted request, we ensure that the request channel has a corresponding Channel record, creating it if needed.

Previously this was done by attempting to insert a new Channel every time, and, if this fails due to the uniqueness constraint (on the image ID and site dictionary) being violated, only then trying to fetch an existing row.

This is correct in the sense that it will always result in the channel being present in the table. The problem is that attempting to insert a channel consumes an id from the underlying sequence that channel IDs are generated from, and rolling back on the constraint violation does not roll back the sequence. So, each request consumes a channel ID, whether or not the channel has been seen before.

In October 2022 when I first noticed this problem, we had 681 rows in channel_v3, and the highest-valued channel ID was 61 177 182.

In March 2024 as I reword this commit message, there are 828 channels, and the highest-valued channel ID is 144 135 429.

This is a 32-bit signed integer field, and the protocol came into use around October 2021.

Based on the October 2022 figures, and assuming the number of online Endless OS systems per hour (the upload frequency) remains constant, I estimated that we would exhaust the channel ID sequence after ((2 ** 31) / 61 177 182) = ~35 years from October 2021, so 2056.

But recalculating with the 2024 figures, the estimate is October 2021 + 14.9 years, i.e. some time in 2036. We've created our own 2036 problem! The true date is sooner because the sequence has advanced far beyond 144 135 429.

Address this problem by trying to retrieve the channel, and only attempting to insert it if it is missing. In case of two processes creating the same channel simultaneously, we still re-fetch the channel if insert fails.

This will stop us from consuming channel IDs quite so fast. We can't really plug the gaps without stopping the service & running a very disk-access-intensive update, but we can live with it.

https://phabricator.endlessm.com/T33969

wjt commented 1 year ago

Wrote a test, if it passes here I'll push another PR to see if it fails without the fix!