Fix scoped storage permission issues causing a crash when opening a typst project folder that was not just created by the app (+ manual refresh for big documents + auto refresh onResume for cloud syncing + background preview refresh without freezing UI) #151
Before this fix, creating a folder worked fine, but accessing an already existing folder crashed. The only exception is when we create a folder, and then try to reopen it just after. But once we erase the app's cache, or reinstall it, or create another folder, then trying to open the previously created folder will crash, just like for any other folder.
This is because of scoped storage limitations introduced in Android 10+. An app can freely create a folder, and access all files inside (files are then considered as "media" files, and the folder a media database of resources), but this is only temporary: the access to this "media" folder will be lost once the app requests the access to another folder or is reinstalled or its cache is erased. This is exactly what happened here.
The solution is to go through the hoops of the scoped storage new system to ask specifically to access an already existing folder and then ask for a permission to write in it. The functions all changed and it's very complicated to get it right.
Fortunately, there is the SimpleStorage library. I implented it here only to open already existing Typst folders, the biggest issue currently IMHO.
The same fix should be applied to creating a Typst folder (although it works currently, the folder is created with the improper method, as a "media" collection instead of being just a raw folder of files - in practice it doesn't change much in this case I think but to be more future-proof it would be better to also use SimpleStorage for creation too) ; and also the fix should be applied to opening files. I did not try to open markdown files because I am using another editor for this purpose, but I guess the same issue will arise.
This PR may also be unclean, I did it very roughly and quickly as I am currently lacking time, it's more a proof of concept than a polished implementation, so please feel free to cherry pick and edit as you see fit.
Note that it includes additional edits than the scoped storage fix (all the changes in TypstProjectViewModel.kt were made prior to try to make rustService more robust against crashes or more explicit with log messages, you can keep or remove as you prefer).
/EDIT: Since I aim to use this app to write my whole PhD thesis, I am going to use it extensively on a daily basis. I added a few other features:
another feature that I think is necessary, the ability to disable the automatic preview refresh, and added a manual refresh button in the top toolbar. This allows to work on documents that are bigger than toy examples, otherwise they are way too slow after each typed character since there is a refresh, whereas here we can refresh only when we want.
Note: Also the manual refresh doubles as a way to manually force a refresh after syncing the files (eg with syncthing), whereas before it was necessary to type in the current document to force the refresh (note that it does NOT refresh the edited text, but only the preview and list of files, to refresh the edited text, it's still necessary to reopen the current file, this is done because otherwise it would be surprising for the user that the refresh button may change their whole inputs!).
onResume() is now implemented to auto refresh the content in the text editor when the app regains focus. Essentially this means that when the app in the background or the screen is turned off, and there is a background cloud files synchronization service that updates the files (such as Syncthing or Dropbox), then when the app regains focus, the latest version of the file will be displayed. This avoids overwriting newer changes with an older version by mistake (eases multidevices synchronization).
Note: a better implementation would be an explicit detection of changes in the background (difference between current value in the app versus the file's content) and ask the user what to do, but the implementation I propose is much easier and works for 80% of cases.
renderProjectToSvgs(), which manages the preview refresh for Typst projects, is now launched in a background process and with a mutex lock to avoid multiple calls in parallel which can crash the app (eg, auto refresh on typing, with a user who types fast!). This avoids non toy Typst projects from freezing the whole app during preview refreshes, so now the user can type and even open other files in the project while the preview is smoothly updated in the background. This kind of removes the need for the manual refresh, but I keep it because it can still be useful for those who want to save on battery or for when the files are edited from outside the app.
TODO:
[x] Test all features on a real phone.
[x] Test on Android 10 (API 29) (real phone, ARM64)
[ ] Test on Android 14 (API 34) (emulated) -- could not test because emulator requires compiling the libraries for x86_64
Before this fix, creating a folder worked fine, but accessing an already existing folder crashed. The only exception is when we create a folder, and then try to reopen it just after. But once we erase the app's cache, or reinstall it, or create another folder, then trying to open the previously created folder will crash, just like for any other folder.
This is because of scoped storage limitations introduced in Android 10+. An app can freely create a folder, and access all files inside (files are then considered as "media" files, and the folder a media database of resources), but this is only temporary: the access to this "media" folder will be lost once the app requests the access to another folder or is reinstalled or its cache is erased. This is exactly what happened here.
The solution is to go through the hoops of the scoped storage new system to ask specifically to access an already existing folder and then ask for a permission to write in it. The functions all changed and it's very complicated to get it right.
Fortunately, there is the SimpleStorage library. I implented it here only to open already existing Typst folders, the biggest issue currently IMHO.
The same fix should be applied to creating a Typst folder (although it works currently, the folder is created with the improper method, as a "media" collection instead of being just a raw folder of files - in practice it doesn't change much in this case I think but to be more future-proof it would be better to also use SimpleStorage for creation too) ; and also the fix should be applied to opening files. I did not try to open markdown files because I am using another editor for this purpose, but I guess the same issue will arise.
This PR may also be unclean, I did it very roughly and quickly as I am currently lacking time, it's more a proof of concept than a polished implementation, so please feel free to cherry pick and edit as you see fit.
Note that it includes additional edits than the scoped storage fix (all the changes in TypstProjectViewModel.kt were made prior to try to make rustService more robust against crashes or more explicit with log messages, you can keep or remove as you prefer).
Fixes #149
For those who want to try the debug apk: https://github.com/lrq3000/BeauTyXT/releases/tag/scoped-storage-fix4-and-background-refresh
/EDIT: Since I aim to use this app to write my whole PhD thesis, I am going to use it extensively on a daily basis. I added a few other features:
TODO: