search

Homebrew / homebrew-cask

🍻 A CLI workflow for the administration of macOS applications distributed as binaries
https://brew.sh
BSD 2-Clause "Simplified" License
20.96k stars 10.74k forks source link

finding outdated casks with and without appcast #55268

Closed core-code closed 5 years ago

core-code commented 6 years ago

hello HBC community,

one of the 'problems' of Homebrew Cask is that it requires constant maintainance to keep working. any Cask with a version number needs to be updated every time a new version of this app is released.

for apps that have an sparkle/rss or github/atom feed from the vendor, keeping them up-to-date is relatively straightforward. @brianmorton has a great system implemented here. (his system incidentally also works for non-sparkle appcasts, but currently most of the Casks that have no sparkle, also have no appcast-stanza)

therefore i've been focusing my efforts on automatically finding outdated Casks on those without "sparkle appcast". every day i let a script run that compares the version of the Casks without appcast, to the database of app-versions that our company maintains. every day the script finds around 5 - 10 outdated Casks that should be updated e.g. using 'cask-repair'.

the design and implementation of the system is discussed here: https://github.com/Homebrew/homebrew-cask/issues/52837 and there was also a bit discussion about this here: https://github.com/Homebrew/homebrew-cask/issues/54811

so, long story short, basically it would be great if the community could have a look at the "cr-processed-DATE.txt" files i place here: http://macupdater.net/cask-repair/ each day and look through them to see which of those entries are real, official new releases of the apps and submit cask-repairs accordingly. i think this would also be a good task to get started with contributing to HBC.

for the time being, it would be best to note false positives (i.e. entries that are not real official releases of this app) here so i'll add them to the different filter-lists in the postprocessing script so that they won't appear in the next days output anymore.

UPDATE 11.dec.2018: ok, the cask-repair system for apps without appcast has been working quite well over the past month, i guess around 80% success rate (i.e. percentage of entries in the txt files that are actually valid new releases). but, it seems like there are quite a lot of casks WITH appcast that are outdated despite the appcast. so i've implemented two new checks to try to improve the situation there:

crsparklefeed-DATE.txt => this checks each cask with an valid sparkle appcast each day and generates an output if the appcast has a new 'newest' entry since yesterday. since sparkle feeds should only contain valid releases, 100% of those should be valid updates to be done via 'cask-repair' (if someone else hasn't beaten you to it).

crgithub-DATE.txt => this checks each cask that has its releases on github (as denoted via the URL or APPCAST stanza) and checks github if a new release has been posted since yesterday. if available, it tries to query the 'Latest Release' as denoted on github by the developer, so that beta releases are not picked up. if no 'Latest Release' is marked, it falls back to just using the newest release and downgrades from error to warning.

suschizu commented 5 years ago

2018-12-07 is online 🔨

ran-dall commented 5 years ago

@core-code Done.

phocus - I can't find this anywhere and the download requires a user account.

suschizu commented 5 years ago

@ran-dall thanks! you're the best :) I will have a look on phocus

suschizu commented 5 years ago

2018-12-08 is online

suschizu commented 5 years ago

hey @ran-dall - phocus is up-to-date :)

ran-dall commented 5 years ago

@core-code Done. No problems today. 🎉

core-code commented 5 years ago

@ran-dall thanks for your good work on the repairs! i'll have little time for HBC throughout the rest of the year but @suschizu is handling this nicely now ;)

ran-dall commented 5 years ago

@core-code NP. I'm also running a little light on time for the rest of the year. BUT I've actually automated a lot of the process so it's pretty easy to do. So it doesn't really matter...as long as I get a heads up, everything is peachy...👍

suschizu commented 5 years ago

2018-12-09 is online :)

ran-dall commented 5 years ago

Done. 👍

electron - beta darktable - Release Candidiate

suschizu commented 5 years ago

2018-12-10 is online - just a few updates today :)

suschizu commented 5 years ago

@ran-dall thanks for the info! the electron and darktable - update will be ignored now sorry for the iridium-issue (see #55941) - it would be good for us to have the "real" version number in the version-stanza

ran-dall commented 5 years ago

@suschizu Thanks for keeping me in the loop! 💁‍♂️

The updates have been sent. ✊

Problems:

suschizu commented 5 years ago

2018-12-11 is online

suschizu commented 5 years ago

@ran-dall thanks for your contribution :) sorry for musescore - I thought I've dropped it from the list … that seem to be a beta-release yes.

suschizu commented 5 years ago

18-12-11 we've added a new check-list ( crgithub-2018-12-11.txt ) - the new script looks only for GitHub-releases 🎄

core-code commented 5 years ago

i've updated the original post title and text with explanation about two new checks that should be uploaded to the server daily now

ran-dall commented 5 years ago

@suschizu Thanks for the constant communication! Way to go! ✊

Done.

aseprite - https://github.com/Homebrew/homebrew-cask/pull/55996 nanowallet - https://github.com/Homebrew/homebrew-cask/pull/55989

ran-dall commented 5 years ago

@core-code These updates make sense. 🙂 I updated my automation to support both outputs now. We'll see how they do in the next couple of days. 😁

core-code commented 5 years ago

awesome thanks! also, i've just added something to let you know if a PR is already open for the app in question - hope that helps ;)

suschizu commented 5 years ago

2018-12-12 two new sheets online 🎉 including the brand new crsparklefeed-check :D

ran-dall commented 5 years ago

@suschizu done. 👍

glyphs - https://github.com/Homebrew/homebrew-cask/pull/56040 - This one was acting strange. rightfont - 5.2.3 > 4.12 - It's already at a whole major release higher.

suschizu commented 5 years ago

2018-12-13 all 3 lists online for today - much todo this time 😣

suschizu commented 5 years ago

@ran-dall - you're awesome! Thanks for all your efforts 👍

ran-dall commented 5 years ago

@suschizu done. ✊

boostnote - Pre-release - 0.11.12-pre brave-browser - Developer Channel v0.59.5|v0.59.5 => Beta Channel v0.59.6|v0.59.6 - ?? Never seen this bytecoin - Beta - 3.4.0-beta-20181212 jbrowse - RC - 1.16.0-rc1 nextiva - No newer version available

suschizu commented 5 years ago

@ran-dall - wow … that was fast 😮 - thank you! 👍

core-code commented 5 years ago

we probably need some form of postprocessing for the github output as well if there are so many non-stable "latest releases" in there.

i suspect some developers don't get that the "latest release" in github should point to the latest STABLE one ;p

suschizu commented 5 years ago

2018-12-14 cr-processed,crgithub and crsparklefeed are online - yay 🍻

ran-dall commented 5 years ago

Done

datadog - https://github.com/DataDog/datadog-agent/releases / Not released yet milanote - No newer release taskexplorer - https://objective-see.com/products/changelogs/TaskExplorer.txt / No newer release jcryptool - https://www.cryptool.org/en/jcryptool/143-news/407-release-candidate-8-en / RC gog-galaxy- No newer release media-center - No newer release cloudtv - No newer release teamviewer - @core-code I think this should be removed from our list electron - v3.0.11 => v2.0.15 ??

core-code commented 5 years ago

sorry at the moment i have no way to postprocess or filter the sparkle or github files. will be looking to add that

ran-dall commented 5 years ago

@core-code Sounds good. Let me know if you need any help with that.

Just to note, I had a lot more returned casks today than normal. The ones with 'No Newer Release' I've manually verified to have no newer release on the appcast. There might be a newer version, but as you know, sometimes I just don't have time to look hard enough... 😳

core-code commented 5 years ago

any help with that.

i think we could just reuse the "postprocessCaskRepairOutput.py" and adapt it slightly. i will do that when i have time, but if you beat me to it .. ;)

I had a lot more returned casks today than normal

well, i think we kind of need to treat the 3 checks differently. the 'database' check has been going for about 2 months now, so it its working quite well i believe.

the appcast check should be quite good since in theory we don't need to verify that those are real releases. anything that occurs in a valid appcast is a real release by definition. i had no time yet to veriy that this check really works as well as expected though.

the github check is also new and i think a bit problematic because many projects seem to tag beta releases as the "latest release". this is the one where we should apply some postprocessing to filter releases that are not official. from a quick look this should be quite easy with regular expressions. but we need to monitor it a bit.

ran-dall commented 5 years ago

just reuse the "postprocessCaskRepairOutput.py" and adapt it slightly.

This seems like a really decent idea. I'll look into this weekend. ✊

the appcast check should be quite good

I can confirm that there haven't been any real issues with these unless the URL changes. 👍

github check is also new and i think a bit problematic

I'd also add that some developers have the strangest habits of numbering things. I especially love it 💘 when a developer has a bunch of pre-releases and/or betas but labels them all as releases.

Also, this hasn't had as many problems over the last few days. So at least I'd say we're progressing in the right direction.

suschizu commented 5 years ago

2018-12-14 cr-processed,crgithub and crsparklefeed are online - sorry @ran-dall for all the duplicates and already updated casks!

ran-dall commented 5 years ago

@suschizu It really wasn't that big of a deal today, fortunately. 👍

nextiva - https://www.nextiva.com/support/articles/nextiva-app-mac-setup.html - No newer version on appcast skype - No newer version on appcast wondershare-video-converter-ultimate - No newer version on website sip - curl: (60) SSL certificate problem: self signed certificate

vitorgalvao commented 5 years ago

This issue is already too long for another person to come in as a good first issue, so I’m removing the label and closing. But please do use it to continue your conversation at will. That’s absolutely fine (and encouraged).

If in the future we need to create another of these issues, don’t hesitate to contact me.

I’m going to unsubscribe from the issue, so @mention me if you need me to see something.

vitorgalvao commented 5 years ago

On second thought, I’ll leave it open a while longer, since the top post is still quite useful. See if anyone else is interested.

core-code commented 5 years ago

@vitorgalvao i agree this issue has gotten quite lengthy and probably not the easiest to jump into as a newcomer. also, @ran-dall is taking great care of all the required cask-repairs at the moment, so it doesn't look like we need additional help currently. i think we are making good progress with keeping everything up-to-date, especially with the new checks introduced a few days ago that check appcasts of sparkle-rss and github-atom kind although there have been a few setbacks with submitted beta versions.

initially i thought our output would be a good starting point for the community to jump in an keep the project up to date, which would suggest it would be best to keep this issue open. however, it seems like @ran-dall is fully on top of things especially with the help of automation, so keeping the issue open may not be necessary or even helpful. so, i defer to both of you on this question. if things change, we can always re-evaluate.

vitorgalvao commented 5 years ago

Then lets keep with the plan of closing, and if things change, we can always make a new one in the future. Thank you all!

suschizu commented 5 years ago

2018-12-16 cr-processed,crgithub & crsparklefeed ready to check 🔨

ran-dall commented 5 years ago

@suschizu Thank you! I'll process these in a few hours when I get back.✊

@core-code @vitorgalvao Sorry gentlemen for not being around yesterday. 🤪 My wife is making me go to the middle of nowhere for X-mas 🎅 and I've been running around like a crazy person. I'll spend some time later looking over things in detail. I'll also start adding some specs for the Machine Learning / "AI" update bot (nothing yet is a 'real' AI, but for lack of a better word).

core-code commented 5 years ago

no problem - if you want us to do today's cask-repairs, just us know, we can always jump in ;)

ran-dall commented 5 years ago

@core-code Thanks for offering but I think I should be good today. However, on the 18th, I'll be travelling. ✈️ If you or @suschizu could take care of it that day. I would be very grateful. 🙏 I loaded up the system into my MacBook Pro this morning with a friend of mine, so I could be able to pick it back up the 19th. I'm off to run today's work. ✊

core-code commented 5 years ago

no problem, we will pick up tuesday

ran-dall commented 5 years ago

@suschizu @core-code Crazy day today! Anyway's here's the works.

ran-dall commented 5 years ago

The details of my automation will come sometime in the next few days when I have time to write it out. Sorry for the delay. 😪

core-code commented 5 years ago

i'll have a look at datadog-agent.

taskexplorer this seems a beta, should be ignored jcryptool this seems to be a rc, should be ignored

i think we prefer 'real' appcasts, i.e. either sparkle RSS or github ATOM feeds, or whatever configuration file the app uses for its own a built-in update check. but if those aren't available, we can use whatever URL contains the latest version in some way, e.g. the download or release-notes website. so, you are right, the download site of jcryptool should be added as an appcast, i've done it now, thanks.

suschizu commented 5 years ago

2018-12-17 cr-processed,crgithub & crsparklefeed are online 👼

suschizu commented 5 years ago

@ran-dall - no problem I'll check the lists in the next few days :) Have a nice holiday and a merry Christmas! ☃️ 🎅 🎄 yes - the datadog-agent is special (https://github.com/Homebrew/homebrew-cask/pull/54739)

ran-dall commented 5 years ago

@suschizu Awesome! 👍 It's only tomorrow. But I'll keep the offer in mind because I might need your help on the 28th when I'm supposed to be travelling back. ✈️

👉 I'll start to process the casks for today...