Key on EOS: app does not work offline

vanessa-chang commented 10 months ago

Follow up from https://github.com/endlessm/eos-image-builder/issues/117

There are some customized images the would need the app content preloaded, so that the user can access it offline.

Content: https://github.com/endlessm/endless-key-content-private/issues/103

AC:

User should be able to access the content either online/offline
User should be able to get the corresponding package based on the locale setting

vanessa-chang commented 10 months ago

Tested an es preload image: https://images.endlessos.org/builds/12693/

There are findings:

It would take ~2 mins to run the download process
If the app is started offline, the download progress bar will sit at 1%. It will not continue after re-connect to an internet.
The Library only show the preloaded channels (ie. 8 channels for the current Spanish starter pack)

dbnicholson commented 10 months ago

I think this is going to be fairly straightforward, at least for the case where we've properly preloaded the expected data. For content nodes, I believe that Kolibri will find that everything is present and the scheduled tasks will become no-ops.

The issue is channel database imports. Since channel databases are versioned, Kolibri always tries to download them to see if there's a new version available. However, the collections packs already contain the desired version of the databases. We can check that we already have that version of the database available and skip creating a task for it.

Skipping content task creation would be harder since we'd have to duplicate all the Kolibri code that determines what content files are needed. It's all pretty hairy as it has to descend the content tree in the channel and apply some heuristics to decide what nodes to include and which files from those nodes are needed. I'm hoping we don't need that.

Now, if the preload is missing data that the pack expects, then I don't know how to handle that nicely. We'd need to decide what to do in the case that initial download tasks are failing because of missing network access. Currently it just loops forever trying to restart the tasks over and over. I'm going to consider that handling out of scope for this task.

dylanmccall commented 10 months ago

It's actually news to me that we're scheduling tasks here in the first place. Looking at 53-ek-content-preload, my understanding is the image builder leaves us with a fully populated database and all those content nodes marked available. So, why is the explore plugin trying to download stuff at all? I could have sworn it didn't do that before, but maybe I just did a bad job testing things with https://github.com/endlessm/endless-key-content-private/issues/97 :b Sorry for my confusion around this. I had assumed it was working a certain way.

Anyway, um, I kind of feel like the explore plugin should not direct the user to the welcome screen if there is content available in the database. That is, if we're creating that whole download state thing for the first time (somewhere inside kolibri_explore_plugin/collectionviews.py), check if there is any content available, and if there is, exit early with "yep, all's good [everything_is_fine.gif]".

Separately, I think it would be nice to consider a separate general-purpose "bulk import from kolibri/content/bulk-import" plugin here, which should be easier to test (and to point upstream eventually). I described that in https://github.com/endlessm/endless-key-content-private/issues/102. The bit we would need for the image builder to EOS route is the "code that runs when Kolibri starts up" in this document: https://docs.google.com/document/d/15Yc0Tc4oB9hAO-nFCJ_gNDIuAvsBLKsE5UVhzarNWtc/edit?skip_itp2_check=true&pli=1.

One nice thing with that approach is it gives us a way (outside of generating a db.sqlite3 file) for the image builder itself to signal to Kolibri that it has some content to start with, so we aren't doing any guessing and we also know exactly what content pack the user is supposed to be seeing.

dbnicholson commented 10 months ago

Anyway, um, I kind of feel like the explore plugin should not direct the user to the welcome screen if there is content available in the database. That is, if we're creating that whole download state thing for the first time (somewhere inside kolibri_explore_plugin/collectionviews.py), check if there is any content available, and if there is, exit early with "yep, all's good [everything_is_fine.gif]".

That's the way it did work until #863, but that's too simplistic. We need to apply tags during initial setup and the image builder can't do that since the pack selection needs to be delayed to runtime. If we just checked if there was content when the tags hadn't been applied, then the discovery page would be empty.

You could maybe optimize to skip to the tagging if any content exists, but you still need to validate that all the necessary nodes are available or the tagging might fail. So, I think it is appropriate that the downloader is run to ensure that you have all the necessary content. We just have to make it smarter to not actually perform any downloads unless really needed.

dylanmccall commented 10 months ago

Okay, in a separate thread (https://github.com/endlessm/kolibri-explore-plugin/issues/898) I'm going to try my hand at the bulk-import stuff that's in my head :) I'm not 100% sure if it'll work out in time so it shouldn't block what you're working on (and those issues you have in mind are 100% worth solving either way). But I'm hoping it'll give us a way to kind of circumvent what we're running into here.

dbnicholson commented 10 months ago

I'm going to make a release so we can consume this in the flatpak and test it out.

dbnicholson commented 10 months ago

This is available to test in 7.11.0 and in the latest flatpak.

vanessa-chang commented 10 months ago

@dbnicholson is there a preloaded image available for the test?

dbnicholson commented 10 months ago

Yeah, https://images.endlessos.org/builds/12861/.

vanessa-chang commented 10 months ago

I tested the image (eosimpact-eos5.0-amd64-amd64.231027-160050.es_GT.img) today, the issue is still there as:

"unable to connect" message is show (I guess it needs a translation too?)
Click retry button, it will continue with the download process, but it will sit at 1% over 5 mins
restart the app, it will still stop at 1%.

Flatpak info: version: 0.7 build date: 2023-10-26

eos-diagnostic-231101_144021_UTC+0800.txt

I will re-open this issue

dbnicholson commented 10 months ago

In the diagnostics:

nov 01 14:37:26 endless endless-key-daemon[3228]: INFO     2023-11-01 14:37:26,886 Enqueuing task {'task': 'kolibri.core.content.tasks.remotechannelimport', 'params': {'channel_id': '359e048230974c8f80db1a95dc80d544', 'channel_name': 'EiE Familias'}} at 1698820646.886678

Hmm, why is it trying to fetch that channel? Shouldn't it already be in the image? Also:

nov 01 14:37:26 endless endless-key-daemon[3228]: INFO     2023-11-01 14:37:26,887 Attempting connections to variations of the URL: https://kolibri-content.endlessos.org

Uh, that should only be used in the image builder. At runtime studio.learningequality.org should be used. Something must be wrong in the image builder code.

dbnicholson commented 10 months ago

"unable to connect" message is show (I guess it needs a translation too?)

This message comes from Kolibri and I think it's a side effect of triggering a channel metadata download without the content server being available. If we need to translate it, it would have to happen upstream.

dbnicholson commented 10 months ago

Oh, wow. If you have any Kolibri option environment variables set when the homedir is first created, they get persisted in the options.ini file. That's... unexpected. I guess the image builder hook should kolibri configure setup first before setting KOLIBRI_CENTRAL_CONTENT_BASE_URL.

dylanmccall commented 10 months ago

Oh, wow. If you have any Kolibri option environment variables set when the homedir is first created, they get persisted in the options.ini file. That's... unexpected. I guess the image builder hook should kolibri configure setup first before setting KOLIBRI_CENTRAL_CONTENT_BASE_URL.

That sounds like a reasonable solution. Alternatively, it should be fine to delete options.ini at the same time we run kolibri manage deprovision. I don't think we need anything other than defaults in there, and Kolibri will happily make a new one whenever it starts.

Although, indeed, I never noticed that it persists environment variables there. Out of curiousity (and hoping that it would be an easy fix), I checked and it's the same behaviour with Kolibri 0.15.