Switch to using Github Release Page for archives

pjrobertson commented 2 years ago

Replace the existing qsapp.com/archives and github.qsapp.com/ pages with GH Releases. Here are the steps we need to do:

[x] Add checksum to GH Releases (@n8henrie )
[x] Upload all previous versions from https://github.qsapp.com/ to GH Releases
[x] Make sure each release that sets a new "min OS version" states it in the Release notes
[x] Redirect qsapp.com/archives to github.com/quicksilver/quicksilver/releases
[x] Delete the gh-pages branch in the main repo and turn of GH Pages integration.
[x] Update Release Process in wiki to remove steps for adding to the 'archives'.

n8henrie commented 2 years ago

Find a way to sign GH Releases

Can you clarify what you mean? 2.0.3 is both codesigned and the tag / release is GPG signed as well (green checkbox):

Can we check that one off?

Upload all previous versions from https://github.qsapp.com/ to GH Releases

It looks like you can cut a release from an existing tag: https://github.com/quicksilver/Quicksilver/releases/new, so it looks like we should be able to select e.g. v1.0.0 and create a release et voila. However, I assume this would mess with efforts like https://github.com/Homebrew/homebrew-cask/pull/121662; I'm pretty sure Homebrew will be relying on the chronological order of a release irrespective of a tag name, and this would bump all Homebrew cask users (likely a nontrivial chunk of the userbase) back to an earlier version. @timvisher is that your understanding? @pjrobertson @skurfer do you think migrating old releases is worth possibly disrupting Homebrew users?

@timvisher you asked about pre-releases -- is there an escape hatch where we could add a suffix like v1.0.0-archive and avoid having Homebrew try to use that version?

The last 3 tasks should be relatively easy!

pjrobertson commented 2 years ago

Can you clarify what you mean? 2.0.3 is both codesigned and the tag / release is GPG signed as well (green checkbox):

Sorry, I think I meant a checksums. I saw somewhere you said you wanted to find a way to add MD5 checksums for homebrew or something.

n8henrie commented 2 years ago

Ah, makes sense!

timvisher commented 2 years ago

@n8henrie I wasn't actually familiar with this particular piece of the machinery in Homebrew but I did some digging and I think I can answer these questions now.

I think your concern about the backporting of older releases to the GitHub release page may be unfounded. If I'm reading the code right, what this strategy actually does is visit the releases/latest redirect URL for a given project and takes whatever it finds there. IIUC, the docs indicate that the date of the release is taken from the commit it was generated from rather than the time the release was published. It also automatically ignores 'draft' or 'pre-release' releases so you shouldn't even need to do anything funky with naming there to avoid getting picked up so long as you're properly tagging the release as a pre-release or draft.

Finally, if I'm wrong about any of the above I think it would be as simple as burning a patch version to get people back on to the 'real' latest. But I'm pretty sure that should be unnecessary.

That all sound right to you?

n8henrie commented 2 years ago

https://docs.github.com/en/rest/reference/releases#get-the-latest-release

The latest release is the most recent non-prerelease, non-draft release, sorted by the created_at attribute. The created_at attribute is the date of the commit used for the release, and not the date when the release was drafted or published.

Yup, did not know that. Thanks!

Busy few days at work, next week I'll punch a release for 1.0.0 and we'll see how it goes.

Thanks again for the digging!

n8henrie commented 2 years ago

Well, the good news is that creating https://github.com/quicksilver/Quicksilver/releases/tag/v1.0.0 was quite easy, and doesn't seem to have messed up the ordering of https://github.com/quicksilver/Quicksilver/releases (as noted by @timvisher).

The bad news is that the GitHub Action obvious didn't exist in the repo at that commit, so it didn't run and doesn't provide a dmg (only a tarfile and zip).

But it looks like I can manually attach something later.

n8henrie commented 2 years ago

Hmmm, trying to batch download so I can manually upload, and there are a lot of 404s here.

$ BASE='https://github.qsapp.com'; curl -s "${BASE}" |
    xmllint 2>/dev/null --html --xpath '//a[starts-with(@href, "downloads/Quicksilver")]/@href' - |
    awk -F'"' -v BASE="${BASE}" '{ print BASE "/" $2 }' |
    parallel 'curl -I -s -w "%{url} :: %{http_code}\n" -o /dev/null' |
    sort -k3
https://github.qsapp.com/downloads/Quicksilver%201.0.0.dmg :: 200
https://github.qsapp.com/downloads/Quicksilver%201.1.0.dmg :: 200
https://github.qsapp.com/downloads/Quicksilver%201.1.1.dmg :: 200
https://github.qsapp.com/downloads/Quicksilver%201.1.2.dmg :: 200
https://github.qsapp.com/downloads/Quicksilver%201.1.3.dmg :: 200
https://github.qsapp.com/downloads/Quicksilver%201.2.0.dmg :: 200
https://github.qsapp.com/downloads/Quicksilver%201.2.1.dmg :: 200
https://github.qsapp.com/downloads/Quicksilver%201.2.2.dmg :: 200
https://github.qsapp.com/downloads/Quicksilver%201.2.2.dmg :: 200
https://github.qsapp.com/downloads/Quicksilver%201.3.0.dmg :: 200
https://github.qsapp.com/downloads/Quicksilver%201.3.1.dmg :: 200
https://github.qsapp.com/downloads/Quicksilver%202.0.0.dmg :: 200
https://github.qsapp.com/downloads/Quicksilver%20B36.dmg :: 200
https://github.qsapp.com/downloads/Quicksilver%20B54.dmg :: 200
https://github.qsapp.com/downloads/Quicksilver%20B59.dmg :: 200
https://github.qsapp.com/downloads/Quicksilver%20B60.dmg :: 200
https://github.qsapp.com/downloads/Quicksilver%20B61.dmg :: 200
https://github.qsapp.com/downloads/Quicksilver%20B62.dmg :: 200
https://github.qsapp.com/downloads/Quicksilver%20B63.dmg :: 200
https://github.qsapp.com/downloads/Quicksilver%20B65.dmg :: 200
https://github.qsapp.com/downloads/Quicksilver%20B66.dmg :: 200
https://github.qsapp.com/downloads/Quicksilver%20B67.dmg :: 200
https://github.qsapp.com/downloads/Quicksilver%20B68.dmg :: 200
https://github.qsapp.com/downloads/Quicksilver%20B69.dmg :: 200
https://github.qsapp.com/downloads/Quicksilver%20B70.dmg :: 200
https://github.qsapp.com/downloads/Quicksilver%20B71.dmg :: 200
https://github.qsapp.com/downloads/Quicksilver_B64.dmg :: 200
https://github.qsapp.com/downloads/Quicksilver%201.3.2.dmg :: 404
https://github.qsapp.com/downloads/Quicksilver%201.3.3.dmg :: 404
https://github.qsapp.com/downloads/Quicksilver%201.3.4.dmg :: 404
https://github.qsapp.com/downloads/Quicksilver%201.4.0.dmg :: 404
https://github.qsapp.com/downloads/Quicksilver%201.4.1.dmg :: 404
https://github.qsapp.com/downloads/Quicksilver%201.4.2.dmg :: 404
https://github.qsapp.com/downloads/Quicksilver%201.5.1.dmg :: 404
https://github.qsapp.com/downloads/Quicksilver%201.5.2.dmg :: 404
https://github.qsapp.com/downloads/Quicksilver%201.5.3.dmg :: 404
https://github.qsapp.com/downloads/Quicksilver%201.5.4.dmg :: 404
https://github.qsapp.com/downloads/Quicksilver%201.5.5.dmg :: 404
https://github.qsapp.com/downloads/Quicksilver%201.5.6.dmg :: 404
https://github.qsapp.com/downloads/Quicksilver%201.5.7.dmg :: 404
https://github.qsapp.com/downloads/Quicksilver%201.5.8.dmg :: 404
https://github.qsapp.com/downloads/Quicksilver%201.5.9.dmg :: 404
https://github.qsapp.com/downloads/Quicksilver%201.6.0.dmg :: 404
https://github.qsapp.com/downloads/Quicksilver%201.6.1.dmg :: 404

Manually confirming in my web browser that the link for 1.5.9 404s.

n8henrie commented 2 years ago

Also, we have a duplicate

n8henrie commented 2 years ago

Manually uploaded the DMG for https://github.com/quicksilver/Quicksilver/releases/tag/v1.0.0. Pretty easy and seems to have worked, I'll chug away at this.

n8henrie commented 2 years ago

there are a lot of 404s here.

I was expecting a typo or escaping error, but looks like the files just aren't there: https://github.com/quicksilver/Quicksilver/tree/gh-pages/downloads

The effort isn't monumental, but if the page has been relatively unmaintained for so long, I'm questioning the effort of updating the releases page with binaries for every release. Seems certainly worthwhile for "last supported version" for the last couple of MacOS releases, but 1.6.2 vs 1.6.1... ¯\_(ツ)_/¯

Also, when I rsync them from the server they have odd names: com.blacktree.Quicksilver__*.dmg (I assume the server renames the content-disposition upon download). Is there a way to batch resolve the names? If not, maybe we should stand up the old archives pages just so I can scrape the missing DMGs (e.g. 1.6.x would be pretty important and isn't avail at https://github.com/quicksilver/Quicksilver/tree/gh-pages/downloads).

I suppose I could just script something to checkout each tag and build locally, but that would take a while and frankly I'd rather just use the official release files that we've been distributing.

pjrobertson commented 2 years ago

The effort isn't monumental, but if the page has been relatively unmaintained for so long, I'm questioning the effort of updating the releases page with binaries for every release. Seems certainly worthwhile for "last supported version" for the last couple of MacOS releases, but 1.6.2 vs 1.6.1... ¯_(ツ)_/¯

Agreed. Just the last supported versions for each release should be enough. You can find the original DMGs on the update server, if they're not in the GH repo. They're in ~/www/qs0/plugins/files - look for files starting with com.blacktree.Quicksilver__XXXXX.dmg - note that XXXX is the decimal version of the hex string used in QS, e.g. 4030 -> 16432. 😢

pjrobertson commented 2 years ago

I see @n8henrie has made a start on uploading releases to the GH releases page. I've updated https://qsapp.com/download.php so that the 'other versions' now points to https://github.com/quicksilver/Quicksilver/releases

Sidenote: we should make clear on every release which version of macOS it's compatible for, to help people find the version they need.

timvisher commented 2 years ago

Sidenote: we should make clear on every release which version of macOS it's compatible for, to help people find the version they need.

I wonder if this is something that Homebrew Cask can do as well. I think Homebrew has become increasingly aggressive about dropping support for older macOS versions but I bet there's an easy way to pin a release to a particular macOS version set and use that one if you're on an older macOS version. Going to add that to my list of things to explore. :)

n8henrie commented 2 years ago

#!/usr/bin/env bash

set -Eeuf -o pipefail
set -x

cleanup() {
  diskutil unmount /Volumes/Quicksilver
}

trap cleanup EXIT

main() {
  rsync -a 'qs0.qsapp.com:www/qs0/plugins/files/*.dmg' .

  local ver dmg dest

  while read -r dmg; do
    hdiutil mount "${dmg}"
    ver=$(defaults read /Volumes/Quicksilver/Quicksilver.app/Contents/Info.plist CFBundleShortVersionString || true)
    hdiutil unmount /Volumes/Quicksilver/

    # Skip undetected version
    [[ -z "${ver-}" ]] && continue

    # Skip existing from prior runs
    [[ -d "${ver}" ]] && continue

    mkdir -p "${ver}"
    pushd "${ver}"
    # Match the format of GH Releases
    cp "../${dmg}" "Quicksilver.${ver}.dmg"
    shasum --algorithm 256 "Quicksilver.${ver}.dmg" > checksum.txt
    popd

  done < <(find . -maxdepth 1 -iname 'com.blacktree.Quicksilver__*.dmg')
}
main "$@"

EDIT: This clones all the dmgs from the servers and then renames them based on the human-readable string (CFBundleShortVersionString) in the same naming format that is produced by GA.

n8henrie commented 2 years ago

I think the "upload old release dmgs" part is about done; I've uploaded at least one for each 1.x, usually the latest one, and included checksums:

$ curl -s 'https://api.github.com/repos/quicksilver/Quicksilver/releases' | jq -r .[].tag_name
v2.1.0
v2.0.3
v2.0.2
v2.0.1
v2.0.0
1.6.1
v1.6.0
v1.5.9
v1.5.8
v1.5.7
v1.4.2
v1.4.1
v1.3.4
v1.3.3
v1.2.2
v1.2.0-final
v1.1.0
v1.0.1-pre
v1.0.0

A few minor issues:

There wasn't a tag for v1.0.1, so I used the tag for v1.0.1-pre -- which means that the binary uploaded there might not match the source code for that tag. But the binary should be available for end users; just if anybody is worried about reproducible builds, they won't find them here (not in the ones I'm manually uploading at least).
Same for v1.2.0-final
I (just now) manually renamed 1.6.1 -> v1.6.1 to match
There is no dmg available for 1.6.2 (also searched https://github.com/quicksilver/Quicksilver/tree/gh-pages/downloads), so I couldn't upload that one, but it seems like kind of an important one. Does anyone know where that DMG went?
I didn't bother with the old ß versions

I'm not worried about anything other than the 1.6.2 version.

n8henrie commented 2 years ago

Redirect qsapp.com/archives to github.com/quicksilver/quicksilver/releases

@skurfer @pjrobertson -- I think this is reasonable to do now. I assume this needs to be changed at the hosting / DNS level -- can you take are of that? I don't think I have the relevant credentials.

pjrobertson commented 2 years ago

I don't think I have the relevant credentials.

It's in the nginx conf file on the server. You should have access, but I'm happy to make the change if nginx conf files isn't your thing.

I think this is reasonable to do now.

I think there's one vital bit of information missing before we can do that - I added an extra task to the first post here: make sure we clearly document which release works with which version of QS. It will make it easier for people going back and trying to find the relevant QS versions for them

n8henrie commented 2 years ago

make sure we clearly document which release works with which version of QS

Sounds important, though I have no idea how to test! I always run the latest release supported by my hardware, and I'm writing this from my only MacOS device. Based on https://github.qsapp.com/, the only listed requirements I see are:

Quicksilver 2.0.0 - Universal Build, Requires macOS 10.14 (Uploaded: 25th March 2022)   6.2MB
Quicksilver 1.6.0 - Requires macOS 10.12 (Uploaded: 19th September 2018)        6.4MB
Quicksilver 1.5.1 - Requires macOS 10.10 (Uploaded: 24th Dec 2016)      6.3MB

I added the requirements to 2.0.0 and 1.6.0; I didn't find a copy of 1.5.1 so I added the Requires macOS 10.10 to 1.5.7 instead (as noted above, tried to upload the most recent "minor" version I could find, so for 1.5 that was 1.5.7).

n8henrie commented 2 years ago

though I have no idea how to test

https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources

macOS Monterey 12 | macos-12 | macOS 12 is currently in public beta.
macOS Big Sur 11 | macos-latest or macos-11 | The macos-latest label currently uses the macOS 11 runner image.
macOS Catalina 10.15 | macos-10.15 |

So it looks like we could test (EDIT: some versions) in CI if we wanted, though I imagine we need to fix the tests before that would be helpful.

n8henrie commented 2 years ago

It's in the nginx conf file on the server.

Ah, found it, thanks. New to BSD.

pjrobertson commented 2 years ago

Most of the 'requires macOS XXX' info is in the current archives page. It just needs copying over: https://github.qsapp.com/

n8henrie commented 2 years ago

Yup, that's what I went off of as per above. Sorry that likely got lost in the multiple replies.

pjrobertson commented 2 years ago

@n8henrie - I think this can be closed now right? I see all the 'Requires macOS XXX' in there, and the qsapp.com redirecting is working :)

n8henrie commented 2 years ago

I was waiting on confirmation that the "requires macos" info I'd provided was sufficient -- I have not yet changed the nginx conf.

the qsapp.com redirecting is working

Not for me? https://qsapp.com/archives/ -> https://github.qsapp.com/ which still shows the old page. CDN or maybe my DNS is cached somewhere?

pjrobertson commented 2 years ago

OK, I've changed the nginx conf and fixed the redirect.

I was waiting on confirmation that the "requires macos" info I'd provided was sufficient -- I have not yet changed the nginx conf.

What you've said looks sufficient! If you haven't already made the changes, then go ahead and do so.

Only a few outstanding tasks here!

n8henrie commented 2 years ago

Update Release Process in wiki to remove steps for adding to the 'archives'.

It looks like you completed this one as well -- thanks!

I deleted the gh-pages branch.

I edited the nginx config to try to 301 redirect github.qsapp.com -> https://github.com/quicksilver/Quicksilver/releases/. The conf seems to validate and I restarted nginx, but it doesn't seem to be working. Would someone mind taking a look? Maybe TTL problem, it still seems to be giving GitHub's 404 page.

Make sure each release that sets a new "min OS version" states it in the Release notes

This final task -- is there something actionable here? Or can we close this?

pjrobertson commented 2 years ago

This final task -- is there something actionable here? Or can we close this?

I don't think so, perhaps just adding that to the 'QS Release process' in the Wiki, which I've done.

pjrobertson commented 2 years ago

Closed! :)

skurfer commented 2 years ago

I know I’ve been too busy to contribute much (even to discussions), but stuff like this is great. Thanks!

quicksilver / Quicksilver

Switch to using Github Release Page for archives #2727