Discuss new catalog architecture

[kelson42]

The catalogue of the cardshop provides apps and content which can be installed on Kiwix Hotspot cards. Currently this is:

• static and code-driven for the apps (shared between Kiwix Hotspot code and the Cardshop)
• dynamic and data-driven for the ZIM files

This model was good enough so far but will be highly challenged because:

• the number of apps we want to manage tends to increase and the containerisation of the architecture makes this pretty easy
• we need other sources of content if we want to introduce new LMS softwares (like for moodle or Kolibri fpr example)

The current system will soon start to be pretty cumbersome to maintain and a few things might simply not possible to do.

We need a solution which allows:

• To add/remove/change easily a service in the app catalogue
• To deal with external/non-zim catalogue of content

[kelson42]

My current thoughts regarding a new architecture are:

• YAML driven catalogue with both apps and content sources
• YAML file would be made available online but maintain via git (at least in a first time)
• Apps entries would provide everything necessary: metadata+technical data
• Contents would not be listed but we would have sources of content. Of course each source would have a type (ZIM, Moodle, ....) and their handling would be hardcoded

Interrogation: would helm be appropriate/beneficial to use?

[rgaudin]

Do you mean having a single catalog with both content entries and content-sources entries ? Like:

---
contents:
- id: wikifundi-fr
  kind: app
  name: WikiFundi en français
  compose:
    xxx
- id: files-gcompris 
  kind: files-package
  name: Gcompris
  url: https://drive.offspot.it/files/gcompris-1.0.zip
- id: kiwix-zim 
  kind: zim-opds
  catalog: https://library.kiwix.org/catalog/v2
- id: kiwix-dev-zim 
  kind: zim-opds
  catalog: https://dev.library.kiwix.org/catalog/v2
- id: fo-zim
  kind: zim-opds
  catalog: https://catalogs.kiwix.org/FO/catalog/v2
  require-auth: true
- id: kolibri:
  kind: kolibri
  studio: https://studio.learningequality.org
  require-auth: true

Interrogation: would helm be appropriate/beneficial to use?

helm is software. I don't see how this can help. What's powerful and not k8s-related about it is the values expansion/merging feature. I don't know if it's available separately nor if we'll need it. What do you have in mind?

[kelson42]

Do you mean having a single catalog with both content entries and content-sources entries ? Like:

No, both apps and content sources (like https://download.kiwix.org/library/ideascube.yml for example). The content metadata themselves would have to be then retrieved in a second stage (and dynamically) from the cardshop and/or imager.

Interrogation: would helm be appropriate/beneficial to use?

helm is software. I don't see how this can help. What's powerful and not k8s-related about it is the values expansion/merging feature. I don't know if it's available separately nor if we'll need it. What do you have in mind?

We should probably discuss this directly, but I ask myself if we will need that our apps definition to be templates (so with variables) and then I wonder if we are not reinventing the wheel - helm does that already. Like you see, this is premitive/high level thinking.

[rgaudin]

No, both apps and content sources (like https://download.kiwix.org/library/ideascube.yml for example). The content metadata themselves would have to be then retrieved in a second stage (and dynamically) from the cardshop and/or imager.

I don't get that. ideascube.yml has no content-source (or we're not clear what a content-source is). Only thing different than ZIM catalog is the type: field which I understand we need.

Do you mean having a meta-catalog of content-sources and then dedicated catalogs like the existing Kiwix one for ZIM, a new one for Apps, a new one (or the same) for files ?

Maybe an example of what your catalog(s) could look like would be easier to understand