Custom Backup of SC

[lordfolken]

https://github.com/streetcomplete/StreetComplete/blame/e6124b90ccb0c7c1fd7957708bd900a4aeaaf63c/app/src/main/AndroidManifest.xml#L31

In this commit backup was disabled globally for SC.

While the reasoning is sound of why you don't want certain things backuped. Its annoying to have to reconfigure SC when changing android device.

There seems to be some ways of handling this gracefully: Reference; https://developer.android.com/guide/topics/data/autobackup

"Auto Backup excludes files in directories returned by [getCacheDir()](https://developer.android.com/reference/android/content/Context#getCacheDir()), [getCodeCacheDir()](https://developer.android.com/reference/android/content/Context#getCodeCacheDir()), and [getNoBackupFilesDir()](https://developer.android.com/reference/android/content/Context#getNoBackupFilesDir()). The files saved in these locations are needed only temporarily and are intentionally excluded from backup operations."

Could this be used to isolate caching, changes, uploads from configuration?

[matkoniecz]

See https://github.com/streetcomplete/StreetComplete/commit/611005bd3e4ebd0cffc9e5f117c3920ff13eecdd for relevant commit that disabled backups

[riQQ]

The only thing that makes sense would be not-uploaded-yet answers, but they should not be stored for a lengthy period anyway plus we wouldn't want these answers to be spread over all of the user's devices (cause then there would be upload conflicts)

This is not current anymore, as now there can be multiple presets each with individual sorting.

[mnalis]

This is not current anymore, as now there can be multiple presets each with individual sorting.

True, that is why SCEE has backup/restore feature for settings and quests presets. (in StreetComplete, I copy QR URL for each of the settings and send it to myself as an e-mail. Inelegant workaround, but works).

But I find SCEE approach much preferable to the approach in this issue - not only does it give you much more options (i.e. you can easily backup your settings before you mangle them to test something and restore afterwards), but you may not want to have all of your devices to use the same settings. And still the main point of commit message stands, its feels very kludgy to say "backup everything, except this 99.9% of the things" (as you basically only want to backup the preferences if I read it correctly, right?)

Just the other day on OsmAnd telegram group guy had wasted a lot of time uninstalling and installing OsmAnd only to find out eventually that the problem is that (unwanted) settings were being pulled from google backup and breaking his "fresh" installs...

[lordfolken]

Please dont make single cases part of your policy for the vast majority. People with multiple devices and different needs also generally know how to disable backups of induvidual apps.

I have 150 apps on my phone, of those around 50 have backup disabled. I dont want to reconfigure 50 apps when i change phone, nor does anyone else.

[westnordost]

To answer your question in the commented commit: I don't know. I just switched off a boolean option. I guess there speaks nothing against allowing backup of certain stuff, like settings, non-uploaded answers or maybe even everything. But my last information is that the limit is 25MB which means that it will pretty much always be above that limit so some custom backup policy would be required that cherry-picks only certain data to be back-upped. And I didn't read into that. You or anyone else is welcome to do this and create a PR for that.

[lordfolken]

So I had a look at the datasize in the explorer and my installation had 14mb of user data. Most of that seems to be in the pictures folder. The tiles are already in the cache folder, so shouldn't be concerned by the backup. So in theory one could simply enable backup and things should work as expected.

The question is what happens to the Pictures. Are they stored as long as needed and deleted on upload?

[westnordost]

The tiles are only the background map tiles. The bulk of the data will be the downloaded OSM data which is kept automatically for up to 14 days before being discarded. So, if you go outside and map a bit for a few days in two weeks, I am pretty sure that the data will always be well above 25MB. The downloaded OSM data is (currently) kept in the main database which is not in the cache folder. (Maybe it could be moved... currently, all database tables are in one database.)

[westnordost]

Which pictures do you mean, the avatar pictures? Or the photos taken to attach them to nodes?

[lordfolken]

The Photos taken by the user.

[lordfolken]

Whatsapp and other applications seem to use Android/media/ for storage of such things. I at the moment fail to retrieve any decent description of where android expects data to be.

[westnordost]

The photos taken by the user are connected to the notes / note comments he did not upload yet. So this is definitely not part of the "cache" but must also be part of the backup if the notes / note comments he did not upload yet are also part of the backup.

[thany]

I don't get the fuss, tbh. I like backups, but what does SC so desperately need to backup? Settings? It doesn't take more than a few minutes to set it up again, and if you must, you can copy them over from your old device. On top of that, you're not switching devices every 2 weeks, are you?

As for un-uploaded notes and pictures: don't back them up. When the phone's on wi-fi again, it should upload them, and when you do a backup, you'll likely do that over wi-fi as as well anyway. They are effectively in volatile storage, because they are indeed temporary. Just don't switch devices before having uploaded any pending quest answers?

Yes, profiles are outright the only thing I can imagine are annoying to re-setup. That's why I mentioned you can also just copy over the data folder from your old device. Or even use Google's app+data backup feature if you're not that tech-savvy.

The point is, backups are for data that can never be replaced once lost. Backups for mere convenience are already provided by Android.

[lordfolken]

Yes, profiles are outright the only thing I can imagine are annoying to re-setup. That's why I mentioned you can also just copy over the data folder from your old device. Or even use Google's app+data backup feature if you're not that tech-savvy.

The point is, backups are for data that can never be replaced once lost. Backups for mere convenience are already provided by Android.

The google/android backup is exactly what the manifest is disabling by setting backup: false.

Android Backup or SeedVault in my case, will not backup an application or its data/settings/profiles, when the backup flag in its manifest is set to false.

Copying files around on android as you suggest, is almost impossible for a normal user to accomplish with later android versions 13> thanks to storage application framework / SE Linux and generally not having access to application folders without being root.

[thany]

I wonder why backup is set to false then 🤔

Is there any particular disadvantage to having this app in a Google backup, that is so severe that is had to be disabled for everyone?

[matkoniecz]

the limit is 25MB which means that it will pretty much always be above that limit so some custom backup policy would be required that cherry-picks only certain data to be back-upped. And I didn't read into that. You or anyone else is welcome to do this and create a PR for that.

https://github.com/streetcomplete/StreetComplete/issues/5197#issuecomment-1682191607

[lordfolken]

The main issue, seems to be that the osm data isn't in a cache folder such as android suggests. https://developer.android.com/guide/topics/data/autobackup

[Helium314]

It can't be in a cache folder, because it's in a database ("Files in the directory returned by getDatabasePath(String), which also includes files created with the SQLiteOpenHelper class"). Even if it's possible to move the database to a cache folder, you would lose your profiles, hidden quests and non-uploaded answers whenever the cache is cleared, because they are in the same database.

[westnordost]

Couldn't there be two databases? One in cache, one in ... not in cache?

[Helium314]

Not sure if you can put a database in cache at all, but you can exclude it from backup via some xml. But someone would need to do the work.

[riQQ]

Documenting my research:

Using a database in a non-default directory, e.g. the cache directory, is possible by passing a custom context to StreetCompleteSQLiteOpenHelper (first argument in the snippet below).

https://stackoverflow.com/questions/5332328/sqliteopenhelper-problem-with-fully-qualified-db-path-name

https://github.com/streetcomplete/StreetComplete/blob/4c5267998cb80ea71dba4bbac173cf6a2abe06f8/app/src/main/java/de/westnordost/streetcomplete/data/DbModule.kt#L7-L10

[mnalis]

@riQQ Although I think it would be much preferable to exclude database from backup via some xml as mentioned above instead of putting database in cache.

---

Various vendors have enabled-by-default (and often hard to find/disable, especially if one does not look for it in advance) features that periodically (like, weekly or so; or when device decides it is "low on resources" or whoever knows what heuristics they choose) "optimizes" the devices by (among other things) clearing all the caches.

It would be already quite annoying if moving database to cache would mean that in such cases the areas one has invested effort to download for offline use were to disappear.

But it it would be outright horrible if one has invested whole day solving SC quests and the system decided to clear the caches and in the process nuked all the solved quests before I got home to upload them. Such an experience (even happening once to them) could easily turn off people from using the app ever again. (regardless if it was technically the vendor system app that did it, and not the app code itself -- users would blame the app, as the same problem does not happen in other apps)

[westnordost]

But it it would be outright horrible if one has invested whole day solving SC quests and the system decided to clear the caches and in the process nuked all the solved quests before I got home to upload them.

Obviously such data would not live in the database that is in the cache directory but in the app data directory.

[thibaultmol]

Also, there's a thing now to actually save auth keys securely as well: https://developer.android.com/identity/sign-in/restore-credentials