search

Googlers-Repo / magisk-modules-repo-util

A Util for Magisk Modules Repo
GNU General Public License v3.0
23 stars 6 forks source link

Future plans #1

Open DerGoogler opened 5 months ago

DerGoogler commented 5 months ago

Hey @IzzySoft, is there any plan that you'll use the new MRepo cli? Nothing has really changed — no categories, anti features etc.

What are your future plans?

IzzySoft commented 5 months ago

Thanks for reaching out, @DerGoogler – and apologies for 1) not having reported back earlier and 2) not even having had time to work on this…

With Sanmer's repo-util being archived now, this certainly moved up in priority on my roadmap. Quite busy with a few other tasks (yes, the same as before), so it will still take a little until I can get to this. Until now, apart from being quite busy, one thing holding me back was concerns about "future compatibility" with updates from Sanmer – concerns which are now rather moot.

As you ask, let me ask one thing back: Will you keep this very repo-util here maintained? Then I'd switch to your fork here – which would be much easier and faster than starting with an entirely different repo-builder, and thus could probably be accomplished faster and earlier. I have no ETA yet, but the "main task" keeping me occupied at the moment will hopefully be accomplished in a few weeks, giving me a break to take care for switching to your fork here. I might even find an earlier slot then, given it would probably work as drop-in replacements and the new fields could be added to the modules one-by-one later.

DerGoogler commented 5 months ago

The fork is stable. Fields like antifeatures, icon are placed inside tracks.json

This is how a module looks like

{
      "id": "acp",
      "name": "Audio Compatibility Patch",
      "version": "v2.5",
      "versionCode": 39,
      "author": "zackptg5, ahrion, John Fawkes",
      "description": "Fixes music and streaming apps (Spotify, Pandora, etc) that aren't processing audio effects for various equalizer applications through the modification of audio policy",
      "added": 1445340430.7142262,
/*
     "lastupdated": "2023-12-28",
     "lastupdated_ts": 1703786679,
     "lastbuild": "2023-12-28",
     "added_ts": 1703840636.160768,
*/
      "timestamp": 1643646579.0,
      "track": {
        "type": "GIT",
        "added": 1445340430.7142262,
        "license": "",
        "homepage": "",
        "source": "https://github.com/Magisk-Modules-Repo/acp.git",
        "support": "https://github.com/Magisk-Modules-Repo/acp/issues",
        "donate": "",
        "verified": false,
        "cover": "",
        "icon": "",
        "screenshots": [],
        "category": "",
        "categories": [
          "Audio Enhancements",
          "Advanced Audio Mods",
          "App Additions and Features"
        ],
        "readme": "https://raw.githubusercontent.com/Magisk-Modules-Repo/acp/master/README.md",
        "require": [
          "aml"
        ],
        "antifeatures": []
      },
      "versions": [
        {
          "timestamp": 1643646579.0,
          "version": "v2.5",
          "versionCode": 39,
          "zipUrl": "https://gr.dergoogler.com/gmr/modules/acp/v2.5_39.zip",
          "changelog": ""
        }
      ]
    }

I would like to add the commented out properties to. Do you agree with that?

IzzySoft commented 5 months ago

Sure! I'm always open to new fields, especially those filled automatically from existing details (like here pulling it up from the separate versions for the summary). My own scripts ("repo browser") should simply ignore fields they don't know about – and the more details are available on request for those browsing the modules e.g. in MMRL, the better! You might remember (or having seen) that I failed with such suggestions on the original repo.

Just wondering: that added_ts differs from added in the above example is not what it finally will be, right?

If you think it makes sense for the MMRL app to also have a "short description" (speaking "fastlane terms" here), you'd run through widely open gates with me. I'd happily implement that on my end then, given time and possibility (there's no such thing with the modules.prop or anything – but I could imagine setting short_description or summary here myself, by "extracting" it from the longer one). Might be helpful with lists. Just a suggestion, not a "request" (as I'm short on time to work on these things myself currently, how could I :stuck_out_tongue_winking_eye:)

DerGoogler commented 5 months ago

Sure! I'm always open to new fields, especially those filled automatically from existing details (like here pulling it up from the separate versions for the summary). My own scripts ("repo browser") should simply ignore fields they don't know about – and the more details are available on request for those browsing the modules e.g. in MMRL, the better! You might remember (or having seen) that I failed with such suggestions on the original repo.

Just wondering: that added_ts differs from added in the above example is not what it finally will be, right?

If you think it makes sense for the MMRL app to also have a "short description" (speaking "fastlane terms" here), you'd run through widely open gates with me. I'd happily implement that on my end then, given time and possibility (there's no such thing with the modules.prop or anything – but I could imagine setting short_description or summary here myself, by "extracting" it from the longer one). Might be helpful with lists. Just a suggestion, not a "request" (as I'm short on time to work on these things myself currently, how could I 😜)

added_ts and added are no difference, same as lastupdated_ts and timestamp

If you want to open a PR, do it!

DerGoogler commented 5 months ago

@IzzySoft. I've updated some things 😄

User can now create a file in their module called common/repo.json:

{
  "support": "str",
  "donate": "str",
  "cover": "str",
  "icon": "str",
  "license": "str",
  "homepage": "str",
  "screenshots": ["array"],
  "category": "str",
  "categories": ["array"],
  "antifeatures": ["array"],
  "require": ["array"]
}

This allows the devs to manage their module without opening any pr. Every property that doesn't exist will be a null value

DerGoogler commented 5 months ago

I've adde a sib command to generate sitemaps

cli.py sitemap --base-url "https://..."
DerGoogler commented 5 months ago

@IzzySoft, the development of MRepo has been completely stopped, even the new repo util.

I don't plan to fork the new repo util, the current does it job perfectly.

IzzySoft commented 3 months ago

As mentioned in the issue that just linked itself here (see above this comment), I'm now switching to your fork (finally :see_no_evil:). Having some questions there:

I'm now in the process of moving the "extra fields" for track.json (which Sanmer didn't want to implement and I thus had placed into iod_track.json) over to the corresponding track.json (will have to duplicate that for a little until I've adjusted my framework for the repo browser), and will also see to fill the other "gaps" (new and empty fields) wherever possible. Checking the modules.json generated, I wonder about some "duplicated fields" there – that is, fields that exist in the "top level" of each module (but stay empty there) as well as in modules[]->track. For example: license. I've set that in track.json and it shows up in track, but the license field parallel to track stays empty. Can I ignore that? Are those "global fields" accidental duplication, or what purpose do they have (guess they're supposed to come from the new repo.json, see below)? I also see the "global" support field has the value of track->source instead of track->support, while there seems to be no "global" support item. Am a little confused now.

    {
      "id": "73sydney.coloursatmod",
      "name": "Colour And Saturation Modifier",
      "version": "1.1",
      "versionCode": 1,
      "author": "adrianmmiller (73Sydney)",
      "description": "A Colour and Saturation Modifier Module - Default Settings for Pixel 6 Pro as per Justarandomguy's post at: https://forum.xda-developers.com/t/root-tune-saturation-white-balance-and-enable-hbm-on-your-pixel.4575403/"
,
      "added": 1703840636.160768,
      "timestamp": 1703786679.0,
      "size": 15144,
      "maxApi": null,
      "minApi": null,
      "category": null,
      "categories": null,
      "icon": null,
      "homepage": null,
      "donate": null,
      "support": "https://gitlab.com/adrian.m.miller/coloursatmodifier",
      "cover": null,
      "screenshots": null,
      "license": null,
      "readme": null,
      "require": null,
      "verified": false,
      "note": null,
      "track": {
        "type": "ONLINE_JSON",
        "added": 1703840636.160768,
        "license": "GPL-2.0-or-later",
        "homepage": "",
        "source": "https://gitlab.com/adrian.m.miller/coloursatmodifier",
        "support": "https://gitlab.com/adrian.m.miller/coloursatmodifier/-/issues",
        "donate": "",
        "verified": false,
        "cover": "",
        "icon": "",
        "screenshots": [],
        "category": "System",
        "categories": [],
        "readme": "",
        "require": [],
        "antifeatures": []
      },

I've only switched locally for now, didn't put it online yet – waiting for your advice concerning this confusion on my end :see_no_evil: After all, it should go live correctly – and not with some "quirks" added by me due to misunderstanding. Will continue local update as soon as I know what to change and how, and push it online once "clean" and "tested" (that my local framework still can deal with it fine – only tested index generation so far).

PS: love you even thought about a "notes" field there :star_struck: Guess that's intended for "maintainer notes"?

And one more question: the readme describes a common/repo.json (you also mention that above) that should be placed into the module's dir. Would that then be modules/moduleName/common/repo.json (why the additional subdir?) – or modules/moduleName/repo.json? And what's the purpose of this extra JSON that those details are not simply included with track.json?

This allows the devs to manage their module without opening any pr.

Ah. So that file would be found in the corresponding module's repo (maintained by the module's author), and needed to be "sync'd" to the Magisk repo using repo-util? How is that sync supposed to happen? I see no field in track.json pointing to a potential repo.json to pull/sync. Further, how are conflicts handled (e.g. a different license being specified in track.json and repo.json) by the client (and supposed to be handled by my repo browser)?

command to generate sitemaps

Will have to look at that later, once the general change is complete, thanks! One step at a time :wink:

(PS: as you're in Germany like I am: we could have a chat on the phone if you like. Contact info can be found in the imprint of my business site. Just give me a call – or send me your number and I call you :wink:)

DerGoogler commented 3 months ago 
     "size": 15144,
      "maxApi": null,
      "minApi": null,
      "category": null,
      "categories": null,
      "icon": null,
      "homepage": null,
      "donate": null,
      "support": "https://gitlab.com/adrian.m.miller/coloursatmodifier",
      "cover": null,
      "screenshots": null,
      "license": null,
      "readme": null,
      "require": null,
      "verified": false,
      "note": null,

And one more question: the readme describes a common/repo.json (you also mention that above) that should be placed into the module's dir. Would that then be modules/moduleName/common/repo.json (why the additional subdir?) – or modules/moduleName/repo.json? And what's the purpose of this extra JSON that those details are not simply included with track.json?

This part can configure user with a file called common/repo.json. modules[]->license, modules[]->readme etc should always preferred over modules[]->track->license etc. common/repo.json is by/on the remote. you don't have to maintain it

PS: love you even thought about a "notes" field there 🤩 Guess that's intended for "maintainer notes"? Available colors are red, blue, green and yellow. It also supports titles

IzzySoft commented 3 months ago

Wait…

This file should be placed in the modules root directory

is maybe ambiguous then. I understood it as a file in the same modules dir as track.json – but reading between the lines, you probably rather mean

This file should be placed in the root directory of the module's repository (and if exists, is fetched together with the ZIP)

Do I get that right? In that case…

modules[]->license, modules[]->readme etc should always preferred over modules[]->track->license etc.

The former would be the ones coming from the module's repo, right? So what if the maintainer of the "index repo" (like me) needs to override something? For example, categories. We might end up with a mess of "custom categories" if multiple module authors make up their own (I've seen the categories devs put with their RFPs at F-Droid). I'm also not sure what note is supposed for then (especially colored ones), as the readme doesn't detail on it – and your comment only mentions what colors are available…

A few more questions if you permit, as those fields are not described in the readme (or in case of category and categories are not even present):

I guess if I just move the fields from my iod_track.json (antifeatures and category, if there's no translation field) over to track.json, the resulting index should be similar to what it was before, except for the new (and empty) fields from common/repo.json? I still wonder why modules[]->support has the value of modules[]->track->source – it cannot come from common/repo.json in the above example as the module's repo does not have that file (nor has mine). So maybe it's copied incorrectly, or should not have been copied at all?

DerGoogler commented 3 months ago

You just showed me that it need some additional work. Like categories can be misused if it is not validated.

Also a additional process to handle relative urls when using cover, icons, screenshots etc.

readme is supposed to show instructions of how module works or something else. Should be a url, I'll implant a relative variant over the days.

require is meant if a module is depending on a other module, like Systemless Node.js requires also Systemless Mkshrc

The repo doesn't support currently translation :(

IzzySoft commented 3 months ago

Thanks! I just took a look at your repo (the modules.json as well as the module's track.jsons). Here's what I gathered so far. If that's correct, and you want me to, I could make a PR to update the Readme with the details:

track.json

other files

additional process to handle relative urls when using cover, icons, screenshots etc.

Yupp, that'd be great! Not sure what you've planned there, but I'd say relative URLs should be handled as relating to the "index repo" then?

The repo doesn't support currently translation :(

No prob, we can skip that (at least for now). I've just grep'd all the iod_track.json files in my repo, not a single module has an URL in there. Unlikely to be used by modules I guess.

DerGoogler commented 3 months ago

verified is just to mark that the module has good quality and is well maintained.

I also opened a new branch. This currently includes whitelisted categories, track fields like, categories are now excluded and now just putted into modules[]->categories no matter if the categories in common/repo.json are changed. track.json can override common/repo.json to prevent spam and abuse.

IzzySoft commented 3 months ago

Thanks, sounds good! How is category handled should only categories be filled? Picks the first one?

DerGoogler commented 3 months ago

Thanks, sounds good! How is category handled should only categories be filled? Picks the first one?

category can also be added from track.json, but I think I'm removing category because categories can be used

IzzySoft commented 3 months ago

Thanks for letting me know (and good I waited) – so I rather fill the array then. You plan to remove category in the index altogether – or have it simply pick the first category? Just asking so I can keep that in mind when updating my framework later.

IzzySoft commented 3 months ago

OK, now updating all the track.json files. Let me know when you'd have time to check an index if everything seems fine. I could attach a locally built one here (would prefer to only "deploy" once confirmed).

IzzySoft commented 3 months ago

All track.json updated, some graphics added, index generation works fine. Ran a sync (which crashed with a known error from the original repo-util; fixed on my end and PR open if you want to adopt the fix), then ran through. Before I push the results to the server, I'll need to update the hint on which client to use (done locally).

You want to test the index before to make sure, or shall I just push it out, @DerGoogler?

DerGoogler commented 3 months ago

yeah @IzzySoft

IzzySoft commented 3 months ago

yeah = push or yeah = review first? :wink:

DerGoogler commented 3 months ago

I'll review that later ;)

IzzySoft commented 3 months ago

OK, before asking again I read that as "push it now, I'll take a look later". Pushed then, new index is live.

I've also opened a PR fixing a minor bug (if some module does not define any versionCode: do not crash, but report that module and then skip/ignore it). I'd offer another one if you'd accept it – to fill in the gaps in the description for track.json with the README. Here's what I would suggest:

Key Attribute Type Description
id required Str Id of Module (in module.prop)
enable required Bool Whether to enable updates
update_to required Str Follow examples below
source optional Str Url of where the source code lives
homepage optional Str URL
readme optional Str URL with e.g. description, instructions
changelog optional Str Markdown or Simple Text (no HTML)
support optional Str URL to issue tracker/support forum
donate optional Str URL to donation page
cover optional Str URL to cover image (featureGraphic)
icon optional Str URL to icon.png (squared, max 512x512 px)
screenshots optional Str[] URLs to screenshots of the module
license optional Str SPDX identifier
antifeatures optional Str[]
category optional Str category the module belongs to (deprecated)
categories optional Str[] array of categories the module belongs to
require optional Str[] list of module_ids this module depends on
verified optional Bool if module has good quality and is well maintained
max_num optional Int Overload MAX_NUM in config.json
versions auto Int how many versions are present (do not touch!)

I'd also ask for 2 additional fields if I might, corresponding to short_description.txt and full_description.txt in Fastlane. The former would override what desc in module.prop gives, the latter go to the module's details page (which in my repo browser does not yet exist, as that field does not exist either).

And btw, discovered 2 modules in my collection which indeed provide a value for translation, as they use e.g. Crowdin to provide localization – so maybe when adding fields, adding this one too?

DerGoogler commented 3 months ago

Any PR is welcome

IzzySoft commented 3 months ago

OK, then I'll prepare one. Though doing that now would conflict with #3 which already changes the same README section. Any ETA when you will have that merged into main, so I can pull it and base my PR on that?

DerGoogler commented 3 months ago

There are currently no plans to merge that branch due private work reasons. Current plan it to complete this here at the end of 2024

DerGoogler commented 3 months ago

I also currently fight around the MMRL and MMRL-CLI. MMRL needs a deploy automation and MMRL-CLI needs many fixes

IzzySoft commented 3 months ago

OK, no prob. If it's OK with you, I'd then simply open a PR against main with my suggestions, and we see from there?

DerGoogler commented 3 months ago

Yes.

IzzySoft commented 3 months ago

Thanks! PR opened.

DerGoogler commented 2 months ago

@IzzySoft, I began to fork MRepo and will use it as the base for MMRL V4 (including the same license and credits)

Current dev builds: https://github.com/DerGoogler/MMRL-V4/releases

Supported repos: https://github.com/DerGoogler/MMRL-V4

IzzySoft commented 1 month ago

Apologies for my delayed answer, I was on vacation… You want me to pick something up or anything? Any action needed on my part at this point?

DerGoogler commented 1 month ago

@IzzySoft you don't have have to do anything. Just wanted to tell you that.

IzzySoft commented 1 month ago

Thanks!

DerGoogler commented 1 month ago

@IzzySoft One thing you can do. You'll need to upgrade the repo util

IzzySoft commented 1 week ago

Apologies again for the delay (seems I'm either on vacation, or drowning in tasks) – done now, and went easy & smooth! Also ran an index update, so new index is online as well.

DerGoogler commented 1 week ago

No worries!

IzzySoft commented 6 days ago

PS, just as another thanks: before you took over, I had occasional crashes on sync (which is why there's still no automated run). Since I switched to your fork, not a single one anymore, so I could finally consider setting up a cron job for regular sync runs and more timely updates :star_struck:

DerGoogler commented 5 days ago

@IzzySoft https://github.com/Googlers-Repo/magisk-modules-repo-util/releases/tag/v2.7.8

IzzySoft commented 5 days ago

Ah! That was already included when I updated, or should I update again now? (checks commits) Oh, the latter then. OK, let me pull… Done, thanks! Up-to-date again here.