update.json? - Githubissues

IzzySoft commented 1 year ago

Could you please add (and maintain) an update.json for automated fetching of new releases? Thanks in advance!

yzyhk904 commented 1 year ago

Do you have any issue for using Fox's Magisk Module Manager? I don't like "update.json" feature.

IzzySoft commented 1 year ago

Thanks for your response! And yes, I do have. For one¹, it doesn't support custom repos. I'm using (actually, maintaining) a module repo which only serves FOSS modules, as I (and several other people) want to stick to FOSS. Actually, "one day" I plan to set up alternatives for fetching modules that do not have update.json – but that won't be possible any time soon, as there are still too many other open issues which have to be solved first. So I can currently only include modules via update.json (or risk missing updates, which I don't want as that would just add to the load).

TL;DR: having update.json would allow me to include your module right away. Not having it will just keep it on the waiting list, where I already have it, until other means of updating have been established here :wink:

¹ _{there are other reasons, but I don't want to go into depth on them here as some of them are rather subjective}

yzyhk904 commented 1 year ago

I think Fox's manager can handle a custom repo. Isn't it enough?

IzzySoft commented 1 year ago

As said, no. FoxMM is a great app (it was me having brought it to F-Droid by the way), but not everyone wants to use it. We love to have a choice :smile:

I see like I don't want to be glued to FoxMM, you don't want to use update.json in your repo here. Not every developer wants (some do not even care). I have to accept that as well, and so I'm checking in parallel for an alternative handling I can implement. As I wrote, that was planned anyway but I'm not sure how long it might take to implement it (you see the number of issues there, waiting to be processed). My hope was that adding an update.json here would let us get your module in quicker :wink:

IzzySoft commented 1 year ago

To give you some more background, first the obvious:

APK size of FoxMMM: 30+ MB. Some low-end devices will have massive issues with that, most likely not only storage-wise but also performance-wise (I'm often testing apps for inclusion at F-Droid, and especially bigger ones I try on low-end devices. A Wiko Sunny 3 running Android Go 8.1 had very noticeable performance issues if APK size exceeded around 15 MB, so will other low-end devices (and that's not just a guess))
APK size of MRepo: less than 3 MB currently. That is factor 10.
FoxMMM has the Tracking anti-feature as it has Sentry enabled by default. MRepo has not.
FoxMMM comes with an EULA you cannot even read without allowing a bunch of trackers in your browser, walk through a consent banner – and then end up in a screen telling you the page cannot be loaded for security reasons ("To protect your security, app.termly.io will not allow Tor Browser to display the page if another site has embedded it. To see this page, you need to open it in a new window.") Clicking the link to "open in a new window" repeats that circle (consent banner) while the URL indicates it's not worth it ("policy-not-found"). And yeah, the only content you get is "Oops! The policy is not here." Well, makes me quite confident to use it. Not. (This part was added when Androidacy took over FoxMMM – which is since when I stopped updating and finally using the app.) Oh, PS: Did you see the "Do Not Sell My Information" button on that EULA page? Implying if I'd not press it… urn… :thinking:

Next point is I'm coming from a F/LOSS perspective here. For "us F/LOSS folks" it's not sufficient that a module exists and we can download it – we want to know it's F/LOSS. We want to know what license it's covered by, where we can report issues, where the source code is maintained (and we possibly can contribute). Those details are not covered by the "standard repo format" used by the official "Magisk Modules Repo" or the "Alt Repo" (and thus by FoxMMM). Now take a look at this screenshot, taken from my repo's web front-end:

The license field is already available with MRepo, the other ones (website, source, issue tracker, translation, donate, category) are on the road-map and currently "custom properties" supported by my web front-end only. MRepo supports both, the traditional and the enhanced format for modules.json. Sure, once the specs are "defined", FoxMMM could adapt them if they wanted to – but that still leaves the APK size and the tracking.

TL;DR: I could try to establish a separate update mechanism right away. But that would mean I'd have to adjust that later, once the "backlog" at repo-utils has been reduced – and not only in the code for my web front-end, but in the (now custom work-arond) definitions for each of the modules in my repo. I'd prefer avoiding that and rather process the changes in the "proper order".

That said, I of course accept your decision to not provide an update.json here – simply say "no", and I won't complain. Especially I don't want to cause you any head-aches or disrupt your work-flow. There are plenty more modules queued for addition as I'll have to fetch them in a different way (about 30 on my list currently). Having an update.json would just allow me to add your module today – instead of in a month or so (no ETA, as the back-log is not only on my end).

IzzySoft commented 1 year ago

PS: The EULA question was just clarified after I was able to find the embedded page (here it is). It only applies to installations from Play Store, so :man_shrugging:

IzzySoft commented 1 year ago

OK, it seems that repo-util is capable to create the module ZIP directly from a Git repo. I've just tried that with v1.2.4+1204(I guess the current HEAD is taken), and compared the ZIP to the one attached to the corresponding release here. Structure seems to match – but the variant of post-fs-data.sh in your ZIP does not exist in your repo; is there some commit you forgot to push? The one in your ZIP has a for loop that is in none of the two versions of the git history. Timestamp is one day before the one at the current HEAD. So I'm not sure if the copy created by the repo-util is safe to contribute, or if it would cause some issues for those using it then. Can you please check?

yzyhk904 commented 1 year ago

OK, it seems that repo-util is capable to create the module ZIP directly from a Git repo. I've just tried that with v1.2.4+1204(I guess the current HEAD is taken), and compared the ZIP to the one attached to the corresponding release here. Structure seems to match – but the variant of post-fs-data.sh in your ZIP does not exist in your repo; is there some commit you forgot to push? The one in your ZIP has a for loop that is in none of the two versions of the git history. Timestamp is one day before the one at the current HEAD. So I'm not sure if the copy created by the repo-util is safe to contribute, or if it would cause some issues for those using it then. Can you please check?

The released zip was made from my local repo and works well, but post-fs-data.sh in this repo is old and wrong. I didn't pushed the correct one and have already pushed it now. It was my mistake because the correct one was just copied from another module of mine.

Thanks.

IzzySoft commented 1 year ago

Ah, that explains – thanks, also for fixing the code base that promptly! So if I'd now clone the repo and zip the content, the resulting ZIP should be a fine module, right? Then I'd go and add it to my repo, using the "git clone" method for updating (which would check module.prop if the versionCode was increased, and if so bundle a new ZIP).

IzzySoft commented 1 year ago

Alternatively: if you could keep the v out of versionName (or make the V in releases & released file name lowercase) so the two can be matched, I now have an update mechanism in place that would look up your module.prop and download the matching ZIP on updates.

yzyhk904 commented 1 year ago

Ah, that explains – thanks, also for fixing the code base that promptly! So if I'd now clone the repo and zip the content, the resulting ZIP should be a fine module, right? Then I'd go and add it to my repo, using the "git clone" method for updating (which would check module.prop if the versionCode was increased, and if so bundle a new ZIP).

Released ZIP files were tested here on various devices, but the resulting ZIP wasn't always tested on multiple devices (probably only one device or so). I recommend to take a ZIP from the latest released one.

I hope users read the README carefully. "audio-misc-settings" cannot achieve its potential unless manually reducing jitter as written in the README or using "audio-jitter-silencer".

IzzySoft commented 1 year ago

I recommend to take a ZIP from the latest released one.

I'll gladly do that. But for automated updates to work, I'd either need an update.json here – or the case of the vs matching so I can automatically fetch those by constructing the URL using values from module.prop. Example for the latter:

version=v1.2.4 and the pattern https: //github.com/Magisk-Modules-Alt-Repo/audio-misc-settings/releases/download/%v/magisk-module-audio-misc-%v.zip would allow me to construct a file download URL of https: //github.com/Magisk-Modules-Alt-Repo/audio-misc-settings/releases/download/v1.2.4/magisk-module-audio-misc-v1-2-4.zip – but unfortunately, that URL does not exist as you're using upper-case V there (https: //github.com/Magisk-Modules-Alt-Repo/audio-misc-settings/releases/download/V1.2.4/magisk-module-audio-misc-V1-2-4.zip). 3 possible fixes I can think of:

make the V in the URL lower-case
make the v in module.prop uppercase
leave the v out of version= entirely

Currently I can only automate update checking when creating the ZIP from Git whenever module.prop signals a new release.

I hope users read the README carefully.

I hope they do that for each module they consider installing :see_no_evil: In the future, I hope the Readme will be part of the catalog itself, so it's available prominently right a) on the web front-end and b) inside the repo app (MRepo) – in addition to the repos themselves. Waiting for the corresponding task on the back-end and in the app to be completed.

yzyhk904 commented 1 year ago

I think you can get the latest released zip file by seeking the latest release page.

IzzySoft commented 1 year ago

Yes, manually – but not automated. The idea is, as pointed out above, to make your module available with the repo mentioned. Using the methods available to repo-util, that's currently not possible for the said reasons: I have no way to tell when to fetch what, as none of the methods matches. If only for that one single letter (v). Doing it manually is not really reliable as one either checks to often and gets frustrated, or forgets about it for months and gets shocked :see_no_evil:

If you take a look at the repo URL, you'll see there are now almost 60 FOSS modules in. Looking "a level up" you will note I also maintain a repo with FOSS Android apps, currently serving more than 1.000 apps. If there are only 10 I have to check manually, that already starts to get "messy" :wink: Hence my kind request: could you maybe just leave out the v in module.prop? That would be the easiest solution. Thanks a lot – and apologies for my insisting…

yzyhk904 commented 1 year ago

No, No. you can do by using latest release and seeking the url of the latest module zip file. If you need to get its version code lightly, fetch it from its repo source.

IzzySoft commented 1 year ago

Yes, I can do that manually – repo-util doesn't support that. For that I'd need something to directly match on, as outlined above. Would it really be hard to have the v removed from module.prop? Does something on your end rely on that?

yzyhk904 commented 1 year ago

Don't use "version". It's just a name. "versionCode is the true version to be checked automatically. However, I will use lower case "v" for related items.

IzzySoft commented 1 year ago

Don't use "version". It's just a name. "versionCode is the true version to be checked automatically.

Couldn't agree more :rofl: Unfortunately, that's not how I could build a bridge from module.prop to the ZIP. With multiple thousands of apps I've checked for different purposes (integration with my repos, inclusion requests made at F-Droid.org etc), only very few are using versionCode with their tag names. I've also learned it seems easy to forget one must increase versionCode when making a new release (have such a case about once a week) – some devs even insist it's an evil they don't accept, keeping it on the same value forever to be able to downgrade should it be needed… :man_shrugging:

However, I will use lower case "v" for related items.

:partying_face: Great, thanks! So tag & file name will have a lower-case v from now on (for future releases), enabling me to map the ZIP as https ://github.com/Magisk-Modules-Alt-Repo/audio-misc-settings/releases/download/<version>/magisk-module-audio-misc-<version>.zip? That'd be fantastico! Please let me know when the first such release is ready, so I pick it up and make it available via my repo ASAP. And thanks a lot!!!

yzyhk904 commented 1 year ago

You can automatically make a local update.json with a simple script for any module on the Github. I made a sample: makeUpdateJson.zip

IzzySoft commented 1 year ago

Yes, I'm doing that already with 14 modules not having their own update.json. Guess why I've asked for the lower-case v :wink: Here's an example:

{
    "version": "2.2",
    "versionCode": 9,
    "moduleProp": "https://github.com/HuskyDG/zygisk_proc_monitor/raw/master/magisk-module/module.prop",
    "urlPattern": "https://github.com/HuskyDG/zygisk_proc_monitor/releases/download/v%v/magisk-am-proc-start-tool-v%v.zip",
    "zipUrl": "https://github.com/HuskyDG/zygisk_proc_monitor/releases/download/v2.2/magisk-am-proc-start-tool-v2.2.zip",
    "changelog": ""
}

My updater script checks against the corresponding module.prop, and when versionCode there is higher than here updates zipUrl by copying urlPattern, replacing %c by versionCode and %v by version. But if version is v1.2.3 while the tag name is V1.2.3, that does not work.

I cannot have different routines for each module though, that soon gets hard to maintain. You have a case mismatch, the next one has an extra string in version that's not in the tag name, the 3rd doesn't use attachments at releases, the 4th has it at GitLab … Next thing is, using a local update.json means repo-util will check that and download the ZIP again (because I don't want to fiddle with its internal structures), which is an additional overhead. Also, I had to download the zip for each update check; the module that updates daily is quite rare, so that's a waste of resources.

Better keep it simple and maintainable. So I wait for your next release, which then hopefully has a "matching cased v" – and then I'll set up the proper local update.json for it.

yzyhk904 commented 1 year ago

It's not true. You can do automatically by modifying your codes for any module universally on the Github. The Github knows the latest release tag. You can obtain it from Github's server responses. See my script at the previous response.

IzzySoft commented 1 year ago

As pointed out, I cannot write a specific update script for each module, that gets too messy (also consider that 1) not all modules are hosted at Github, 2) not all used tagged releases, 3) some always stick to pre-release which releases/latest will never point to, etc). Further, your script requires 4 wget calls, while mine is just 1 – and yes, I do mind the networking overhead, a.o. for ecological reasons. IMHO keeping the case of that v in sync is the much easier and much better approach :wink:

Thanks for offering the script, though!

yzyhk904 commented 1 year ago

I updated my Magisk modules for Magisk 26.0's new magic mount feature. IMHP your approach isn't promising. Module file names vary often and for any reason.

Edit: my improved script makeUpdateJson-2.zip

IzzySoft commented 1 year ago

As I wrote:

/releases/latest does not include pre-releases. So modules using pre-releases would not be covered
modules are not only hosted at Github
too much network overhead

repo-util offers to use either update.json at the project, checking against module.prop and (if versionCode was increased) clone the repo and zip it up, or using a local update.json. My "local variant" works as described above and thus covers more than just Github or Github releases, doing its job fine so far – until now, no module file names changed (which is part of my pre-check btw, if the ZIP attached to releases is using a consistent patttern); should that be the case one day, my updater should inform me (as downloading the non-existing ZIP will fail :see_no_evil:).

I don't want to add more complexity to that (at least not at this time), but will keep your script here should it be needed. Thanks a lot for your efforts, much appreciated! And special thanks for the "synced v" – I've just added your module now, so it will go into the catalog with the sync later today.

Magisk-Modules-Alt-Repo / audio-misc-settings

update.json? #13