Retention history configuration

[LINKIWI]

My primary motivation for hosting my own sync server is to work around the default limited retention history; see bug: https://bugzilla.mozilla.org/show_bug.cgi?id=985930

I figure that if I host the service and storage myself I can disable this behavior so that everything is retained forever and never deleted.

However, I'm not able to find any references in documentation or source code about the retention policy. How can I configure my sync server instance to keep everything, forever?

[rfk]

The limit on the amount of history synced is mostly a client behaviour, see e.g. hard-coded limits here:

https://dxr.mozilla.org/mozilla-central/rev/c291143e24019097d087f9307e59b49facaf90cb/services/sync/modules/constants.js#77

The client also sets a time-to-live on the history records it uploads to the server:

https://dxr.mozilla.org/mozilla-central/source/services/sync/modules/engines/history.js#7

In theory, you could modify the server to ignore the client-requested TTL, e.g. by dropping it on the floor before writing out the record to the database here:

https://github.com/mozilla-services/server-syncstorage/blob/master/syncstorage/storage/sql/__init__.py#L696

But given that the behavior is mostly controlled by the client, may not actually produce the experience you're after.

[thomcc]

If you're interested, I think we'd accept a patch that moved MAX_HISTORY_UPLOAD (and probably the TTL too) from constants to prefs. (Note that MAX_HISTORY_DOWNLOAD looks related but isn't, it's how many records we download at a time, not the total limit).

[LINKIWI]

That's great! Is there a bug link I can track for its status and its target release cycle?

[thomcc]

Ah, no, you misunderstood me. There's no bug at the moment, or plan to do this work (It's certainly not involved work, but we're busy with the effort of rewriting the future clients and server in Rust).

Basically, if you want prefs for these, we'd be willing to accept a patch implementing them, if you wrote it (this sounds condescending, maybe, but it wasn't intended that way, there's plenty of things we wouldn't accept patches for). Anyway, it's probably only a handful of lines of JS code which would be relatively straightforward to do, and we could point you in the right direction if you're interested.

Sorry for the confusion.

[LINKIWI]

I see, if you're willing to give me some pointers on how to get started I'll see if I can get a patch up when I have the time. As a new contributor I don't have any familiarity with the source tree.

[khimaros]

i believe the area of code @thomcc was referring to is now https://github.com/mozilla-services/server-syncstorage/blob/02d617d655da361fddaa757b92fdd6c8ad46572c/syncstorage/storage/sql/__init__.py#L733

[khimaros]

good news! poking through the syncserver code, it seems the provided docker image will never purge records by default.

from what i can see, the code that removes those bso items https://github.com/mozilla-services/server-syncstorage/blob/02d617d655da361fddaa757b92fdd6c8ad46572c/syncstorage/storage/sql/__init__.py#L776 is only called in the purgettl script here https://github.com/mozilla-services/server-syncstorage/blob/02d617d655da361fddaa757b92fdd6c8ad46572c/syncstorage/scripts/purgettl.py#L28 and i don't see any way for that script to be invoked in the provided docker images.

a quick scan of te syncserver.db confirms this:

sqlite> select strftime('%s000', 'now') as now, modified + ttl as expire from bso inner join collections on collection = collectionid where now >= expire and collections.name = 'history' order by expire limit 10;
1622950261000|1614173667857
1622950261000|1614173667857
1622950261000|1614173667857
1622950261000|1614173667857
1622950261000|1614173667857
1622950261000|1614173667857
1622950261000|1614173667857
1622950261000|1614173667857
1622950261000|1614173667857
1622950261000|1614173667857

perhaps @thomcc can correct me, but it seems the tl;dr here is that if you want unlimited history, use the syncserver docker image!