[request] additional properties for 'track.json' and 'modules.json'

[IzzySoft]

Can we have a website property in track.json, to link to the module's website for further information? A property to point to the module's issue tracker would also be nice. Both properties should be optional, or at least allow for empty values when a corresponding URL is not available.

In this context: how would the util deal with

• additional properties I'd add to track.json which it doesn't know itself (e.g. a comment property)
• additional files I'd place in the module's directory

Would they simply be ignored – or would they be removed? I might need some"control data" for automation/maintenance tasks not covered by repo-util and think about where to place them.

EDIT: details of the requested properties in this comment below.

[SanmerDev]

I think it is necessary, MRepo also needs it.

[IzzySoft]

An answering par of my questions myself: additional files in a module's subdir remain untouched. Just not sure how --remove-module will handle it. Use it currently to place an icon.png inside if the module's repo (or website) offers one for it (preview).

Fields which would come in handy as well include shortdesc and a possibility to override the auto-evaluated name and maybe even description (see e.g. "itsy bitsy" or "wifi only mode" in the preview for what I mean).

[SanmerDev]

• We can add various fields to track.json. As long as the core fields are not modified, this will not affect sync.
• --remove-module will directly delete the directory of the module.
• There seems to be a error with your-repo's repo_url, which makes files inaccessible.

[IzzySoft]

We can add various fields to track.json. As long as the core fields are not modified, this will not affect sync.

So especially, those fields will ot be removed when you rewrite the file?

--remove-module will directly delete the directory of the module.

:+1:

There seems to be a error with your-repo's repo_url

Ouch, a copy-pasta-error – thanks for the hint! Fixed. Good thing I didn't yet publicly announce it :see_no_evil:

[IzzySoft]

You might wish to check the preview again: I finally completed the "backend class" to make access compatible to the one used by my F-Droid repo. Now one can sort and filter :smiley:

While doing that I noticed which other details would be nice to have. In addition to above mentioned website and tracker, in track.json:

• category
• maybe also antiFeatures? Not sure if we'll need them.

In modules.json:

• repo URL & description
• for each module: added, last_update (timestamp); category; versions (from update.json), and if set up with track.json, also AntiFeatures

With those available I could also offer to include other repos with my web front-end, as I do for F-Droid repos – so you could have a WebIF for yours as well. Further it could be nice to have a "full description" (in addition to the summary now set up as "description") to offer more details – which could e.g. be included with modules.json if a corresponding (Markdown? HTML?) file is found in the module's directory. What do you think?

[IzzySoft]

Btw: do you have an account on the Fediverse somewhere (Mastodon, Misskey, Lemmy, Friendica …)? I'm preparing for go-live of my MM-repo, see my first poll on the topic here :wink: When lifting the cover, I plan also to promote MRepo. So if you have a Fedi-acccount, it would be natural for me to mention that.

[IzzySoft]

Here's an example ("summing-up") of useful fields that (with the exception of max_num) should go to track.json (and from there be transferred to the corresponding entry in modules.json):

{
 "web": "https://example.org/",
 "source": "https://github.com/johndoe/mod_magisk",
 "tracker": "https://github.com/johndoe/mod_magisk/issues",
 "translation": "https://example.crowdin.com/translate",
 "donation": "https://example.org/donation",
 "category": "Internet",
 "antiFeatures": ["NonFreeDep","NonFreeAssets"],
 "max_num": 2
}

• web: website of the module (or the author)
• source: where to find the module's source
• tracker: the issue tracker for the module
• translation: where to contribute to translations (usually Weblate, Crowdin, Transifex links)
• donation: website where one can spend the author a coffee (as you know: programmers convert coffee into code :wink:)
• category: which category the module belongs in. Up to the repo owner to define which categories to cater; one could roughly orient at F-Droid Categories
  => maybe this should be categories and an array, to allow for more than one (as F-Droid has it)?
• antiFeatures: if you accept this as optional property, should be dealt with similar to the previous point – see F-Droid's AntiFeatures
• max_num: see #8

Which fields you feel to make optional (i.e. can be empty or even missing altogether) I fully leave up to you. I see that with the fields already present, you prefer them to be there even if empty (see e.g. changelog), which I guess is totally fine;it might be a minor issue, though, when updating an existing installation with a later version of repo-util bringing new fields not yet present with exiting modules in the repo, if you know what I mean. So code should be able to deal with "missing fields".

[SanmerDev]

• Fedi-acccount: Is Telegram ok?, for MReo I have channel and group
• track.json: I don't think we need translation, other fields are acceptable.

Thank you for your attention and publicity to MRepo. I believe that the next version of MRepo will have a huge improvement.😋

[IzzySoft]

• Telegram is not part of the Fediverse, so interaction won't be possible. Well, if you have no Fedi account I'll simply link to your app. Expect 2 new users while I'm writing this (I found some testers to give my repo a run :smiley:)
• No strong feelings about translation (the other fields are much more important to me), though it would be nice to have: some users want to help but are no developers, so they look out to e.g. help with localization (I personally know some of them).
• what do you think of category: will it be a string (with one category), or rather be named categories and be an array? Just asking to have my back-end prepared properly.

Thank you for your attention and publicity to MRepo. I believe that the next version of MRepo will have a huge improvement. :yum:

Gladly – and can't wait to see it happen :smiley: Please give me a ping here for when things become available in repo-util (especially with track.json and modules.json) so I can take the proper steps in time. Of course only in a way to not break MRepo itself – I shouldn't update my repo if it breaks the client :zany_face:

[SanmerDev]

I'll simply link to your app.

It' s Ok

translation, some users want to help but are no developers, so they look out to e.g. help with localization

It seems that few modules provide something like Crowdin to help localization, if the module is used as an extension of the app, it is possible, but this is indeed rare, so we don't need it, and we still have web, source, tracker.

what do you think of category

I think MRepo needs its own category, but I'm not sure how to do that, it's up for discussion then create a docs/categories.md. AntiFeatures, we can use F-Droid's rules and also create docs/anti_features.md. And I wish they all use a string (a Short identifier similar to SPDX ID).

[IzzySoft]

Translation:

It seems that few modules provide something like Crowdin to help localization

At least one of them is in my repo already, yes. I find it useful as it animates users to contribute. But as said, no strong feelings, so I leave the decision to you.

Categories: in F-Droid, an app can have 1..n categories assigned (1 is mandatory, additional ones are optional). Most Magisk modules would just have "System", true, just a few match things like "Internet" (like the proxy module in my repo). But it cannot hurt to be prepared, making it categories and an array, though currently very few modules would need more than one. As for which categories to have: we could orient on the list I've linked above from F-Droid. We could also simply leave it to the "repo operator", but some orientation would be good especially with multiple repos included in the client. So maybe take the F-Droid list here and "strongly suggest" using those categories – while not enforcing them as there might be specific ones needed?

AntiFeatures: Yes, the F-Droid definition uses string identifiers (CamelCased, no spaces allowed). See the names in the linked list. Examples: NonFreeDep (depends on non-free things like GSF being present), NonFreeNet (depends on a non-free network service). You can pick my antifeatures.json](https://gitlab.com/IzzyOnDroid/repo/-/blob/master/lib/antifeatures.json) if you wish. One AF that will probably be added soon is NonFreeComp (contains a non-free component – only needed by 3rd party repos as F-Droid wouldn't let such apps in) – which would be useful here as well (e.g. SafetyNet stuff might need it).

[IzzySoft]

As it seems it will take a little while until you implement this, can we already agree on the structure & fields used – so I can (partly) prepare on my end? Especially:

• will it be multiple categories (array) or just a single category (string)
• similar for donation (as there could be multiple ways, e.g. ko-fi plus paypal)
• as for donations, I guess you'll stick with URLs (F-Droid also allows e.g. IDs for OpenCollective, Liberapay etc, and has fields for Bitcoin/Litecoin)

If the time-gap becomes too large, I might forget what "work-arounds" I placed with what in mind – so I need to comment them accordingly for later replacement when structures are implemented here :wink:

So as per my above comment, can you already say how it will look like in track.json and modules.json?

[SanmerDev]

Due to personal reasons, I can't deal with these problems in time, I should start to solve these issues soon.

[IzzySoft]

Thanks! No pressure meant, just asking for orientation so I can properly prepare things on my end. Once clean, I'll also make my part public under a F/LOSS license with reference to repo-util here, so you can count it a "complementary module" :wink: So once you know, a note will be very welcome (even if implementation is still ongoing then).

[SanmerDev]

New track.json

{
  "id": "zygisk_lsposed",
  "update_to": "https://lsposed.github.io/LSPosed/release/zygisk.json",
  "license": "GPL-3.0",
  "website": "https://lsposed.org/",
  "source": "https://github.com/LSPosed/LSPosed.git",
  "tracker": "https://github.com/LSPosed/LSPosed/issues",
  "donate": "",
  "added": 1679025505.129431,
  "last_update": 1673882223.0,
  "versions": 1
}

New modules.json

{
  "name": "Sanmer Magisk Repo",
  "metadata": {
    "version": 1,
    "timestamp": 1690465277.11947
  },
  "modules": [
    {
      "id": "zygisk_lsposed",
      "name": "Zygisk - LSPosed",
      "version": "v1.8.6 (6712)",
      "versionCode": 6712,
      "author": "LSPosed Developers",
      "description": "Another enhanced implementation of Xposed Framework. Supports Android 8.1 ~ 13. Requires Magisk 24.0+ and Zygisk enabled.",
      "latest": {
        "zipUrl": "https://ya0211.github.io/magisk-modules-repo/modules/zygisk_lsposed/v1.8.6_(6712)_6712.zip",
        "changelog": "https://ya0211.github.io/magisk-modules-repo/modules/zygisk_lsposed/v1.8.6_(6712)_6712.md"
      },
      "versions": [
        {
          "timestamp": 1673882223.0,
          "version": "v1.8.6 (6712)",
          "versionCode": 6712,
          "zipUrl": "https://ya0211.github.io/magisk-modules-repo/modules/zygisk_lsposed/v1.8.6_(6712)_6712.zip",
          "changelog": "https://ya0211.github.io/magisk-modules-repo/modules/zygisk_lsposed/v1.8.6_(6712)_6712.md"
        }
      ],
      "track": {
        "type": "ONLINE_JSON",
        "license": "GPL-3.0",
        "website": "https://lsposed.org/",
        "source": "https://github.com/LSPosed/LSPosed.git",
        "tracker": "https://github.com/LSPosed/LSPosed/issues",
        "donate": "",
        "added": 1679025505.129431
      }
    }
  ]
}

[IzzySoft]

So you do not want to include antifeatures and categories (for the latter, I thought it might also make sense making it an array, as a module might fall into more than one)?

And no per module max_num here? Thought I saw somewhere you've implemented that already… ah, there: #8 (but that was just closed, you indeed didn't say it's implemented – and I cannot find it either). That would be very useful to limit how much space can be consumed, by e.g. restricting larger modules to a lower number of versions kept.

But apart from that: good to have most of the "core data" in the modules.json!

[SanmerDev]

max_num: Now you can set it in track.json (I forgot to write it)

antifeatures and categories: I don't have an idea how to implement them. After all, adding labels is a very heavy work (especially when there are many modules)

[IzzySoft]

max_num: Now you can set it in track.json (I forgot to write it)

Ah :smile: Good.

I don't have an idea how to implement them. After all, adding labels is a very heavy work (especially when there are many modules)

Maybe just having the fields in the JSON already (per module, and also in the index), postponing implementation in the app until decided? I could then cleanup (and finally publish) my corresponding website libraries, which currently still rely on some "dirty hacks" as those fields are not yet available "officially".

If you agree, I think best is making both arrays/lists (more future-proof).

[SanmerDev]

Adding antifeatures and categories fields is simple, the problem is how to classify the modules.

[IzzySoft]

I can help you with that, it's done in my repo already. Examples for antifeatures you can even see with the web browser already (thanks to my "dirty hacks"), simply filter by "AntiFeatures: yes" and then mouse-over the reddish circles next to the module name. You can also filter by category, though due to the details pages still missing (was waiting to have all the data ready first in the index) it's not yet displayed.

If you want to take a look at the index I currently construct (in addition to the modules.json), just download it: it's located right next to the modules.json and named index.json. I basically inject additional details into yours and save that as a second copy :wink:

If you specify what problems you have with classification, I can try answering your questions. I guess it's towards the antifeatures; you can refer to my F-Droid repo for that. I have a JSON with their definitions available, that should give you an idea here.

[SanmerDev]

At present, the antifeatures and categories will not be added.

I will continue to promote the development of MRepo. During this period, there may be some changes in the json structure.

[IzzySoft]

At present, the antifeatures and categories will not be added.

So I still cannot publish my library as it will still depend on "local hacks" :cry:

During this period, there may be some changes in the json structure.

While all this is in motion, I guess it's better I do not update my local branch. Please let me know when things settled and I should give it a try. It would then help to know what changes have been done, so I can adjust my library accordingly. Thanks!

[IzzySoft]

During this period, there may be some changes in the json structure.

While all this is in motion, I guess it's better I do not update my local branch. Please let me know when things settled and I should give it a try. It would then help to know what changes have been done, so I can adjust my library accordingly.

@ya0211 may I kindly ask you not to entirely break the entire thing on such a short notice (and that even only afterwards) again? There were just 14 days between you stating structures are "in flux" ("During this period, there may be some changes in the json structure" on August 5th), and then releasing a new version of MRepo which drops compatibility (August 19th) without any warning ahead, causing a lot of inconveniences for maintainers as well as users of other repositories. I originally had planned to be on vacation since last Friday; in that case things would have stayed broken for 2 more weeks. This is not what I'd call "expected behavior"; not even release notes were giving a hint. Such major steps should be communicated better (especially when explicitly asked for).

Apart from that "big mishap": that was a lot of work on your end, too – thanks for integrating so many of my wishes with the new version! I still need to see that I update the framework on my end to make use of all that, but what I discovered so far looks great.