Upgrade Python version to 3.9 to avoid external storage access issues

rtibbles commented 2 years ago

@manuq reports: External storage errors happen because of https://bugs.python.org/issue28141 which has been fixed in Python 3.9.

manuq commented 2 years ago

Here I have a branch which attempts to use an external storage device for Kolibri data: https://github.com/endlessm/kolibri-installer-android/commit/09c53c98fe4f5f9596a3377f364562d7efba788e

This currently fails at runtime with:

04-21 10:04:31.551 17646 17740 I python  : INFO:root:Initializing Kolibri and running any upgrade routines
04-21 10:04:31.555 17646 17740 I python  : DEBUG:kolibri.plugins.utils:Loaded kolibri plugin: kolibri.plugins.app
04-21 10:04:31.555 17646 17740 I python  : DEBUG:kolibri.plugins.utils:Initializing plugin: kolibri.plugins.app
04-21 10:04:31.941 17646 17740 I python  : INFO     Option CHERRYPY_THREAD_POOL in section [Server] being overridden by environment variable KOLIBRI_CHERRYPY_THREAD_POOL
04-21 10:04:31.943 17646 17740 I python  : INFO     Option DEBUG in section [Server] being overridden by environment variable KOLIBRI_DEBUG
04-21 10:04:31.946 17646 17740 I python  : INFO     Option DEBUG_LOG_DATABASE in section [Server] being overridden by environment variable KOLIBRI_DEBUG_LOG_DATABASE
04-21 10:04:31.950 17646 17740 I python  : INFO     Option RUN_MODE in section [Deployment] being overridden by environment variable KOLIBRI_RUN_MODE
04-21 10:04:32.063 17646 17740 I python  : INFO     Option CHERRYPY_THREAD_POOL in section [Server] being overridden by environment variable KOLIBRI_CHERRYPY_THREAD_POOL
04-21 10:04:32.070 17646 17740 I python  : INFO     Option DEBUG in section [Server] being overridden by environment variable KOLIBRI_DEBUG
04-21 10:04:32.072 17646 17740 I python  : INFO     Option DEBUG_LOG_DATABASE in section [Server] being overridden by environment variable KOLIBRI_DEBUG_LOG_DATABASE
04-21 10:04:32.073 17646 17740 I python  : INFO     Option RUN_MODE in section [Deployment] being overridden by environment variable KOLIBRI_RUN_MODE
04-21 10:04:32.106 17646 17740 I python  : INFO     Attempting to setup using pre-migrated databases
04-21 10:04:32.152 17646 17740 I python  : WARNING  Unable to copy pre-migrated database from /data/data/org.learningequality.Kolibri/files/app/kolibri/dist/home/db.sqlite3 to /storage/D15429D1D0DFFFBDC65168977D6460FF8EC5E06F/Android/data/org.learningequality.Kolibri/files/KOLIBRI_DATA/db.sqlite3
04-21 10:04:32.167 17646 17740 I python  : WARNING  Unable to copy pre-migrated database from /data/data/org.learningequality.Kolibri/files/app/kolibri/dist/home/syncqueue.sqlite3 to /storage/D15429D1D0DFFFBDC65168977D6460FF8EC5E06F/Android/data/org.learningequality.Kolibri/files/KOLIBRI_DATA/syncqueue.sqlite3
04-21 10:04:32.191 17646 17740 I python  : WARNING  Unable to copy pre-migrated database from /data/data/org.learningequality.Kolibri/files/app/kolibri/dist/home/networklocation.sqlite3 to /storage/D15429D1D0DFFFBDC65168977D6460FF8EC5E06F/Android/data/org.learningequality.Kolibri/files/KOLIBRI_DATA/networklocation.sqlite3
04-21 10:04:32.208 17646 17740 I python  : WARNING  Unable to copy pre-migrated database from /data/data/org.learningequality.Kolibri/files/app/kolibri/dist/home/notifications.sqlite3 to /storage/D15429D1D0DFFFBDC65168977D6460FF8EC5E06F/Android/data/org.learningequality.Kolibri/files/KOLIBRI_DATA/notifications.sqlite3
04-21 10:04:32.873 17646 17740 I python  : DEBUG:ifcfg:Distro detected as 'Linux'
04-21 10:04:32.873 17646 17740 I python  : DEBUG:ifcfg:Using '<class 'ifcfg.parser.LinuxParser'>'
04-21 10:04:33.819 17646 17740 I python  : Traceback (most recent call last):
04-21 10:04:33.819 17646 17740 I python  :   File "/sysroot/home/manuq/eos/hack/gitrepos/kolibri-installer-android/src/kolibri/dist/django/db/backends/utils.py", line 62, in execute
04-21 10:04:33.819 17646 17740 I python  :   File "/data/data/org.learningequality.Kolibri/files/app/kolibri/dist/django/db/backends/sqlite3/base.py", line 326, in execute
04-21 10:04:33.824 17646 17740 I python  :     return Database.Cursor.execute(self, query)
04-21 10:04:33.825 17646 17740 I python  : sqlite3.OperationalError: disk I/O error
04-21 10:04:33.825 17646 17740 I python  :
04-21 10:04:33.825 17646 17740 I python  : The above exception was the direct cause of the following exception:
04-21 10:04:33.825 17646 17740 I python  :
04-21 10:04:33.825 17646 17740 I python  : Traceback (most recent call last):
04-21 10:04:33.825 17646 17740 I python  :   File "/sysroot/home/manuq/eos/hack/gitrepos/kolibri-installer-android/src/main.py", line 28, in <module>
04-21 10:04:33.826 17646 17740 I python  :   File "/sysroot/home/manuq/eos/hack/gitrepos/kolibri-installer-android/src/kolibri/utils/main.py", line 276, in initialize
04-21 10:04:33.828 17646 17740 I python  :   File "/sysroot/home/manuq/eos/hack/gitrepos/kolibri-installer-android/src/kolibri/utils/main.py", line 153, in _setup_django
04-21 10:04:33.828 17646 17740 I python  :   File "/sysroot/home/manuq/eos/hack/gitrepos/kolibri-installer-android/src/kolibri/dist/django/__init__.py", line 27, in setup
04-21 10:04:33.830 17646 17740 I python  :   File "/sysroot/home/manuq/eos/hack/gitrepos/kolibri-installer-android/src/kolibri/dist/django/apps/registry.py", line 116, in populate
04-21 10:04:33.830 17646 17740 I python  :   File "/sysroot/home/manuq/eos/hack/gitrepos/kolibri-installer-android/src/kolibri/core/apps.py", line 30, in ready
04-21 10:04:33.832 17646 17740 I python  :   File "/sysroot/home/manuq/eos/hack/gitrepos/kolibri-installer-android/src/kolibri/core/apps.py", line 91, in activate_pragmas_on_start
04-21 10:04:33.832 17646 17740 I python  :   File "/sysroot/home/manuq/eos/hack/gitrepos/kolibri-installer-android/src/kolibri/dist/django/db/backends/utils.py", line 64, in execute
04-21 10:04:33.833 17646 17740 I python  :   File "/sysroot/home/manuq/eos/hack/gitrepos/kolibri-installer-android/src/kolibri/dist/django/db/utils.py", line 94, in __exit__
04-21 10:04:33.834 17646 17740 I python  :   File "/sysroot/home/manuq/eos/hack/gitrepos/kolibri-installer-android/src/kolibri/dist/django/utils/six.py", line 685, in reraise
04-21 10:04:33.834 17646 17740 I python  :   File "/sysroot/home/manuq/eos/hack/gitrepos/kolibri-installer-android/src/kolibri/dist/django/db/backends/utils.py", line 62, in execute
04-21 10:04:33.835 17646 17740 I python  :   File "/data/data/org.learningequality.Kolibri/files/app/kolibri/dist/django/db/backends/sqlite3/base.py", line 326, in execute
04-21 10:04:33.836 17646 17740 I python  :     return Database.Cursor.execute(self, query)
04-21 10:04:33.836 17646 17740 I python  : django.db.utils.OperationalError: disk I/O error
04-21 10:04:33.836 17646 17740 I python  : Python for android ended.

Using adb shell I can see that the files were actually created in the corresponding place:

octopus_cheets:/ $ ls -l /storage/D15429D1D0DFFFBDC65168977D6460FF8EC5E06F/Android/data/org.learningequality.Kolibri/files/KOLIBRI>
total 1128
-rwxrwx--- 1 root sdcard_rw 991232 2022-04-21 10:04 db.sqlite3
-rwxrwx--- 1 root sdcard_rw  32768 2022-04-21 12:34 db.sqlite3-shm
-rwxrwx--- 1 root sdcard_rw      0 2022-04-21 10:04 db.sqlite3-wal
drwxrwx--- 1 root sdcard_rw      0 2022-04-21 10:04 logs
-rwxrwx--- 1 root sdcard_rw  40960 2022-04-21 10:04 networklocation.sqlite3
-rwxrwx--- 1 root sdcard_rw  36864 2022-04-21 10:04 notifications.sqlite3
-rwxrwx--- 1 root sdcard_rw   7141 2022-04-21 10:04 options.ini
-rwxrwx--- 1 root sdcard_rw   1125 2022-04-21 12:34 plugins.json
drwxrwx--- 1 root sdcard_rw      0 2022-04-21 10:04 sessions
-rwxrwx--- 1 root sdcard_rw  40960 2022-04-21 10:04 syncqueue.sqlite3

How to test

Build and install an APK from this branch https://github.com/endlessm/kolibri-installer-android/commit/09c53c98fe4f5f9596a3377f364562d7efba788e
Insert an SD card or a USB stick. I tried different formats, mostly NTFS.
Enable the external storage devices from Settings. It is a bit surprising but at least Chromebooks require this, they are not plug & play:
Also in Settings give the Storage permission in the Kolibri app. This happens automatically if you install using adb install -g. This will have to be prompted to the user at runtime in the future:
Launch the app.

Other things I've tried

Previously in this repo there was a shutil.move() call to move preseeded data to KOLIBRI_DATA. That was failing with "[Errno 1] Operation not permitted" so I was suspecting this Python bug: https://bugs.python.org/issue28141

So I tried almost all the ways to copy files from Python and found that:

shutil.copytree(..., copy_function=shutil.copyfile) was giving the same "[Errno 1] Operation not permitted" as shutil.move
shutil.copy, shutil.copy2 and shutil.copystat for a single file was also giving the same error.
shutil.copyfile works, which doesn't copy file metadata or file permissions.

Things to try next

I need to make a "hello world" APK with python-for-android using Python 3.9 and see if I can reproduce the bug.

manuq commented 2 years ago

I tried with Python 3.9.12 by changing the requirements in the .p4a file:

diff --git a/.p4a b/.p4a
index 1c11471..6b7fd8b 100644
--- a/.p4a
+++ b/.p4a
@@ -5,7 +5,7 @@
 --dist_name "kolibri"
 --private "src"
 --orientation sensor
---requirements python3,android,pyjnius,genericndkbuild,sqlite3,cryptography,twisted,attrs,bcrypt,service_identity,pyasn1,pyasn1_modules,pyopenssl,openssl,six
+--requirements python3==3.9.12,hostpython3==3.9.12,android,pyjnius,genericndkbuild,sqlite3,cryptography,twisted,attrs,bcrypt,service_identity,pyasn1,pyasn1_modules,pyopenssl,openssl,six
 --android-api 30
 --minsdk 21
 --permission WRITE_EXTERNAL_STORAGE

Then I cleared the cache with p4a clean-all and rebuilt the apk. I confirmed that I'm now at 3.9.12 by logging sys.version. I'm still getting the "Unable to copy pre-migrated database" warnings and then the I/O error trying to access the database as in the log above. And just like before the files are being created. So I will continue checking the storage permissions.

manuq commented 2 years ago

I confirmed that this is not about permissions. We already have the WRITE_EXTERNAL_STORAGE declared in the .p4a file. And I'm granting that permission at installation time with adb install -g.

I have opened a bug for python-for-android about the file copy issue: https://github.com/kivy/python-for-android/issues/2589

In any case, the blocker now is the sqlite3.OperationalError: disk I/O error when Kolibri tries to execute a query to the SQLite database. Because I was able to workaround the file copy by patching Kolibri https://github.com/learningequality/kolibri-installer-android/commit/8571918cf38bbeb179cf08cbec65f129f2e43a05 .

rtibbles commented 2 years ago

Fixed in #115

learningequality / kolibri-installer-android