Can't synchronize project with syncthing or other means when using QField 2.0.14

[arkanoid87]

My QField app upgraded automatically from 1.10.0 to 2.0.14 via Play Store.

I've immediately realized that my configured project in shared folder was no more there, and what "Import Project folder/from ZIP" really does is copy the existing project into an internal/private QField app folder that's not reachable from other applications.

I was using Syncthing like suggested here: https://qfield.org/docs/fieldwork/projects.html

Expected behavior:

The modified data.gpkg is reachable from other Android applications

Observed behavior:

The data.gpkg is reachable only from within QField app that doesn't let to synchronize it by other means than USB cable or (possibly) QField cloud (that's still in beta and not ready for production use)

QField version: QField 2.0.14 (8f9aa8)

Additional information:

• Problem started happening recently, didn't happen in an older version of QField: Yes
• Problem can be reliably reproduced, doesn't happen randomly: Yes
• Problem happens with all files and projects, not only some files or projects: Yes

[gdt]

It sounds like you are suffering from the new Android storage rules, which has pushed many apps' data into internal/private storage.

[arkanoid87]

I'm on a stock Samsung device running Andoid 11, not a fringe device/Android version.

Considering that Android 12 is out there, I might say that QField 2.x is a breaking change on 1.x general system architecture.

Synchthing or other custom synchronization methods should be available to the user, not because QfieldCloud is bad, but because existing systems are running and android software updates should not introduce breaking changes

[jfmoyen]

Sadly, it’s not Qfield 2 that is broken. It’s android 11+. Have you read this? https://www.opengis.ch/fr/2022/03/05/qfield-users-sit-down-we-need-to-talk-about-storage-access-on-android%ef%bf%bc/ Qfield 1 was published before the Big Change and as such was immune from it. Qfield 2 was forced to comply. Yes, it is massively annoying.

[arkanoid87]

Thanks for the reply. I know the timeline of events at Google, but on the other hand I still don't understand why QField is blocking other options.

I'm using different apps capable of reading and writing to other app private data right now (on Android 12, just upgraded from 11+). For example I know that open source app KeePassDX published on Play Store and recently updated reads and write to other apps private space like Dropbox or Termux home: https://github.com/Kunzisoft/KeePassDX

Both KeePassDX and Termux are on the Play Store, are open source and have been updated after November 2021. Just install KeePassDX and create a new DB in Termux or Dropbox home and try reading and writing to it.

On the other hand QField even if capable of reading from Termux or Dropbox just like KeePassDX copies those files internally to never give them back except via QFieldCloud or USB.

[gdt]

Another data point, perhaps helpful, is that Vespucci uses Download/Vespucci as a data folder, which is hacky, but lets users do what they want. i can then use syncthing to get tracks and osm files to my computer, and to get geojson back to vespucci.

[jfmoyen]

I'm using different apps capable of reading and writing to other app private data right now

Indeed, Google offers the possibility to allow specific apps to access all folders on the device. In the blog post I was linking, the QField developers explain that they did apply for this possibility, but google denied them this coveted status (presumably because it is not a "core" functionality of the app). Obviously keepass and termux made a better case of explaining why they need full access (and truth be said, I can sort of see the point).

You may as well take it from the other side, and blame SyncThing for not having full access to QField folder ! After all, SyncThing being a backup/sync app would have a much better case for obtaining full access status from Google, and this would solve your problem (which happens to be mine as well...). Except that FieldSync developers explicitly said that they will not go for it. Since SyncThing does bi-directional sync-ing, it would all to easy for a user to play around with files on one device and mess up the behavior of an unrelated app on another device. FS don't want to be in a situation where they may have to deal with irate users having accidentally (and remotely !) deleted something critical...

Vespucci uses Download/Vespucci as a data folder,

I was wondering about that, too. I'm not an Android developer so totally unfamiliar with the file system, but surely, there must be public folders accessible by all apps ? Doesn't Android have a /home directory that belongs to the user and is fully accessible ? Using /Download is rather hacky, for sure....

[arkanoid87]

This is the relevant documentation:

https://support.google.com/googleplay/android-developer/answer/10467955

If QField doesn't fit into google review process policy for this, I think it's better to go the hacky way then and sync from what's left in world writable condition.

Maybe Vespucci is using the Media Store API https://developer.android.com/training/data-storage/shared/media on Downloads folder

[gdt]

I am mostly going to skip assiging blame as not useful. From the highest level, the problem is proprietary software that acts contrary to the user's interest, which leads to all sorts of sandboxing restrictions to control it. But this problem is being faced by a huge number of apps.

Certainly QField is not something that should have Manage Storage permissions, so I won't pursue that.

There is not really a home directory in the unix sense. Each app is a separate uid as part of the isolation design. Within shared storage, there is a Storage Access Framework to get access while managing permissions. A lot of the point is to avoid random apps exfiltrating random data. I think syncthing does have storage manager permissions, and anything in /storage/emulated/0 can be accessed, except for app-private stuff.

So the question is how to have qfield's data in non-app-private storage. Apparently Download is special and an app can obtain lasting pemission to access it. The other option is an app-specific directory that is within the global storage, and OsmAnd can do this. OsmAnd's goals are basically the same here: let people use syncthing/etc. to get tmaps and racks off and put maps on.

Yes, Download/QField is hacky, but it has worked well for me with Vespucci.

[arkanoid87]

On Aug 13 2021 Vespucci went this way

Change public directory to Download/Vespucci and copy files there https://github.com/MarcusWolschon/osmeditor4android/commit/0a6523196196df03955a7161ce9bdb6f354a6acb

Pre-Android 11 targeting migration https://github.com/MarcusWolschon/osmeditor4android/pull/1436

Storage access changes for Android 11 https://github.com/MarcusWolschon/osmeditor4android/issues/1278

Quoting from Vespucci author @simonpoole: https://github.com/MarcusWolschon/osmeditor4android/issues/1278#issuecomment-855236763

Originally I had assumed that the best course of action would be to move the contents of the "Vespucci" directory to the app specific directory in Android/data (that would have been Android/data/de,blau,android), as that directory continues to be accessible by the app and retains UNIXy File semantics and APIs.

However in their deep wisdom, unfathomable for mere mortals, google has made anything in the "Android" directory inaccessible to the system file picker in Android 11. Yes, that means that the app cannot use the system file picker to select files for reading / writing in it's "own" directory. But not only that, even an app with the new file management permissions cannot access such a directory (including dedicated file manager apps). This rules out storing anything there that might need to be selected via an user interaction. And this gotcha already applies to everything running on 11 regardless of if the app is targeting Android 11 or not.

Google doesn't offer a clear alternative storage location for non-media files (after all you are supposed to be a drone non-stop consuming youtube and not doing anything else), but storing these in the "Download" directory may work short term, that is till google closes that loophole (storing data there contradicts the whole reasoning on why the app directories have had to become unusable).

[jfmoyen]

Yes, and it undoubtedly solves the immediate problem. I can understand why a developer would be a bit reluctant to go this route, though. The moment you are effectively using part of a system (here /download directory) for something else than its intended purpose, you put yourself at the mercy of sudden changes in implementation, as you have no guarantee that the hack or quirk you are relying on will be durable. What happens, for instance, if Google decides that /Download being by definition a temporary storage folder everything older than a month will be automatically cleaned from there ? Or, perhaps even worse, if some of the manufacturer-specific software (bloatware ?) decides to take the same measure as part of its cleaning routine ? Or you trigger a somewhat too eager third party cleaning app and don't notice that you did not uncheck "remove old downloads" (which at face value sounds innocuous enough...) ?

It would therefore solve our immediate problem, but we may find ourselves facing the same one anytime in the future - and worse, in an unpredictable way. So, an interim measure, at best.

[jfmoyen]

PS- maybe an interim measure is better than no measure at all, though...

[arkanoid87]

Google documentation says: https://developer.android.com/training/data-storage/use-cases#sharing-non-media-files

Export non-media files to a device

Define a proper default location to store non-media files. Allow users to export files from app-specific directories to a more generally accessible location. Use MediaStore's downloads or document collections to export non-media files to the device. Note: To avoid cluttering, use generally accessible locations like externalStoragePublicDirectory() or [externalMediaDirs()](https://developer.android.com/reference/android/content/Context#getExternalMediaDirs()).

[gdt]

You can't reason about "what if google does arbitrary bizzarre unhelpful thing X" unless you want to stop using Android...

I tend to think the storage-visible per-app dir is best.

[jfmoyen]

You can't reason about "what if google does arbitrary bizzarre unhelpful thing X" unless you want to stop using Android...

Don't tempt me :-)

What I'm saying is rather "you can not rely on something that happens to work coincidentally this month but has no guarantee to still work if something, even unrelated, changes in the system". There are part of the features that you can expect to be stable, and some that you know aren't. They may not change for a period of time if you are lucky. Or not.

[arkanoid87]

Mind the Can other apps access? https://developer.android.com/training/data-storage

seems the only possible way to automate writing is to use the MediaStore API

https://developer.android.com/training/data-storage/shared/media#scoped_storage_enabled

Don't unnecessarily request storage-related permissions for devices that run Android 10 or higher. Your app can contribute to well-defined media collections, including the MediaStore.Downloads collection, without requesting any storage-related permissions.

[simonpoole]

The problems for apps that manipulate larger amounts of data have been a long time coming see a 2019 blog post https://www.openstreetmap.org/user/SimonPoole/diary and insofar are not a surprise.

Using the app specific directory is an option if you don't need users to select files there or are using your own file selector (and the files don't need to be persisted over app de/re-installs).

[jfmoyen]

indeed. Well, I'm not a developer myself, not a QField developer certainly so I was merely discussing my understanding of the situation....

[arkanoid87]

I've double checked that 2.0.10 - beta 10 (first released v2 asset after last stable 1.10.0) still had the "classic" file editing in selected folder, and it has been released Dec 6 2021

From 2.0.11 - beta 11 we have the revamped thing that ruined my monday

[nirvn]

@arkanoid87 , first, let's start with this: we feel your pain. This new storage paradigm imposed by Google is disruptive, and while we're trying to mitigate as much as we possibly can on our end, we're aware of its impact.

We've just finished adding new functionalities to QField (to be released in the coming days) that easily allows for users to export projects and datasets out of QField into a storage location that can then be accessed by synchronization tools like Syncthing. Please give these APKs a try, we're hoping it'll make workflows like yours a lot easier to adapt: https://github.com/opengisch/QField/pull/2733#issuecomment-1099094087

You'll also notice the possibility to compress project folders and send the resulting ZIP archive to 3rd party apps (such as Google Drive, Dropbox, NextCloud, Gmail, etc.). That's an interesting new functionality that might be worth exploring too.

I'll try to address other points you've raised in the comments above, hopefully I'll catch them all :wink:

Don't unnecessarily request storage-related permissions for devices that run Android 10 or higher. Your app can contribute to well-defined media collections, including the MediaStore.Downloads collection, without requesting any storage-related permissions.

This is actually slightly misleading. First, these request-free locations are only for known media file extensions for the most part (i.e. .jpg, not .qgs or .gpkg), and although requests are not needed, the app still requires to access those files through the Scoped Storage API, which is not supported by several underlying libraries used by QField (to name a few: Qt and gdal). So your hint at "well simply use the download directory mate!" wouldn't work ;)

Google doesn't offer a clear alternative storage location for non-media files (after all you are supposed to be a drone non-stop consuming youtube and not doing anything else), but storing these in the "Download" directory may work short term, that is till google closes that loophole (storing data there contradicts the whole reasoning on why the app directories have had to become unusable).

As the developer himself note, this Download directory hackish workaround is not something that's future-proof, and is foreseeable something Google will shutdown too. In any case, this still requires accessing file content through storage APIs which aren't compatible with the libraries QField relies on.

Thanks for the reply. I know the timeline of events at Google, but on the other hand I still don't understand why QField is blocking other options. I'm using different apps capable of reading and writing to other app private data right now (on Android 12, just upgraded from 11+). For example ...

As others have said here, rest assured QField isn't trying to "block" you. The new storage restrictions are imposed by Google, not by individual app developers. As for mentioning other apps, it is the case that Google will whitelist apps in a completely arbitrary fashion. We've tried and failed to convince Google in the past; we will continue to argue our users should be given the opportunity to opt in all files access, but ultimately the decision is not in our hands.

I think that covers the major points raised here. We do hope these new export capabilities will be able to unlock workflows for you that will be satisfactory.

[Dubhain]

@nirvn

I mentioned this on the blog, but if this is the way the wind is blowing at Google, then wouldn't publishing on F-Droid be better? Would you not be unrestricted then as to what permissions you use? Without sounding too political (I hope) Google get away playing God in situations like this only because the alternatives are not adopted, not because they don't exist. I may just not understand the way the limitations work, but reading the documentation that @arkanoid87 posted, it sounds like it's to do with publishing on Google Play, not Android itself - so just don't publish on Google Play?

@arkanoid87 Just to mention my current solution, in case you hadn't thought of it. I've downloaded the old apk from here and turned off automatic updates on Google Play. Works perfectly for me.

[github-actions[bot]]

The QField project highly values your report and would love to see it addressed. However, this issue has been left in feedback mode for the last 14 days and is being automatically marked as "stale". If you would like to continue with this issue, please provide any missing information or answer any open questions. If you could resolve the issue yourself meanwhile, please leave a note for future readers with the same problem and close the issue. In case you should have any uncertainty, please leave a comment and we will be happy to help you proceed with this issue. If there is no further activity on this issue, it will be closed in a week.