Correct way to upgrade from syncserver to syncstorage-rs

[psychofaktory]

I am currently using the Docker version of the Firefox Syncserver behind an nginx proxyand would now like to migrate to the new Syncstorage-rs.

What is the right way to switch over, transfer the existing data and not have to modify the configuration on all clients?

┆Issue is synchronized with this Jira Task

[jackyzy823]

Currently no: See comments and issues: https://github.com/mozilla-services/syncstorage-rs/issues/1051#issuecomment-924885375 and https://github.com/mozilla-services/syncstorage-rs/issues/1051#issuecomment-924973833

https://github.com/mozilla-services/syncstorage-rs/issues/1495

[jrconlin]

Going to add a few points here as well.

The Syncserver is not the source of truth about your data. You browsers are. That's why it's best to have multiple instances (User Agents) of Firefox connected to Sync which exchange data. Sync is just there to make sure that all of the user agents are eventually consistent.

Each type of data that is synced is called a "collection", and how each collection is handled can be a bit different. Not all collections are complete (e.g. your "history" only goes back so many entries). Some collections only provide references (e.g. "tabs" don't get copied and opened on other devices, but the information about them is shared between user agents).

What that means is that as you switch things over to the new sync system, the user agents will populate it with the data that they already have, and eventually things will become consistent between the various instances. Just realize that any changes on user agents that have not been migrated to the new server won't appear on the user agents that have been migrated.

[aidanhs]

Am I reading this correctly that there being no upgrade path means that, in the course of the upgrade, the 'official' firefox sync instance has dropped all data from old synced clients under the assumption they will be resynced?

This is quite sad :( I had some very old clients I no longer had access to (dead computers) and on attempting to use a firefox sync cli to get tabs from those machines it no longer works... I kinda just assumed there would be no reason the data would ever need to vanish (short of shutdown of the sync service entirely).

[jrconlin]

Unfortunately, Sync was never created to be a backup tool. It may act like one, but ultimately the goal was just keeping data up-to-date between instances of Firefox. (Less than fun fact: when we ran our own servers, they would fail at a terrifying rate. That's one of the reasons we have the Tokenserver, which routes users to a new storage server. Running your own server can be more reliable because you're not handling anywhere near the level of traffic and disk activity.)

Somewhat highlighting this is the fact that sync storage only contains the latest data set. When you change your password, all the old data becomes inaccessible, and should be dropped. The "sync" part of Sync should merge the most recent data with older data, but even then, it tends to favor the more recent. I'll add that even with the move to more "durable" sync, this is still very much the case. Running sync on a distributed database just means that our server failure rate is no longer counted in days and there's far less of a chance that data disappears. It's still possible, of course. Systems can and will fail, but the likelihood is far, far less.

I am a bit curious about the data for the dead computers. Was this data tied to a different account? If you still have that account and password, then it should (technically) be recoverable to a new instance of Firefox that's pointing to that data store. Once the sync is done, you could then either back up that profile's information or export it.

If you don't have the account or password anymore then, yeah, that data is no longer accessible, regardless of whether or not it's still in your database.

[jackyzy823]

Unfortunately, Sync was never created to be a backup tool.

Please ,please don't say so. https://support.mozilla.org/en-US/kb/switching-devices

[psychofaktory]

If I summarize what @jrconlin says correctly, a direct data transfer from syncserver to syncstorage-rs is not required at all. As long as the previously registered Firefox agents connect to the new syncstorage-rs, as the data will also be transferred.

The point that is not yet clear to me: With the syncserver, the path to be entered on the clients was /token/1.0/sync/1.5. For syncstorage-rs, the corresponding path is /1.0/sync/1.5. Unfortunately, I do not have the possibility to manually adjust the path on all relevant endpoints.

Would it be sufficient in this case to simply set up a syncstorage-rs server, point my nginx reverse proxy to the new instance instead of the old one and have the path adjusted via a simple URL rewrite rule?

[jackyzy823]

Would it be sufficient in this case to simply set up a syncstorage-rs server, point my nginx reverse proxy to the new instance instead of the old one and have the path adjusted via a simple URL rewrite rule?

Yes but needs some tricks.

server {
        ## storageserver part
    location / {
        proxy_pass http://syncserver:5000;
        ## Important for calculating Hawk auth (requiring host and port).
        ## Refer: https://docs.rs/actix-web/latest/actix_web/dev/struct.ConnectionInfo.html
        ## https://github.com/mozilla-services/syncstorage-rs/blob/9916b3bdb0506e9805f505007222f189f1c4dc54/syncserver/src/web/auth.rs#L173-L197
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $host;
    }

        ## tokenserver part
    #! To be compatible with Python self-hosted version's endpoint
    #! For not changing anything in client side
    location /token/ {
                ## slash is important
        proxy_pass http://syncserver:5000/;
                ## not required by tokenserver, but no harm.
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $host;
    }
}