Automate KOReader Updates

[gbyl]

I wrote the script below to push KOReader updates. It uses github cli gh and assumes the GNU implementations of date, awk, and sed. It has worked well enough for the past 3 releases. However, there must be a more robust/elegant solution from the Toltec maintainer's point of view. Any ideas @matteodelabre @Eeems?

#! /bin/bash

version=$(curl -so - https://ota.koreader.rocks/koreader-remarkable-latest-stable.zsync | awk -F '[v.]' 'NR==2 {print $2"."$3}')
wget -q --show-progress "https://build.koreader.rocks/download/stable/v$version/koreader-remarkable-v$version.zip"
time_stamp=$(date -u -r koreader-remarkable-v$version.zip +%Y-%m-%dT%H:%M:%SZ)
checksum=$(sha256sum koreader-remarkable-v$version.zip | awk '{print $1}')
branch_new=koreader-$version
toltec_path=$HOME/toltec

cd "$toltec_path" || exit 1
git switch testing
git pull upstream testing
git branch $branch_new
git switch $branch_new
sed -i -e "s/pkgver=.*/pkgver=$version-1/" -e "s/timestamp=.*/timestamp=$time_stamp/" -e "s/[a-z0-9]\{64\}/$checksum/" ./package/koreader/package
git diff --unified=0
git add .
git commit -m "Update KOReader to $version"
git push -u origin $branch_new
git branch -v
gh pr create --head $(git config user.name):$branch_new --base testing --repo toltec-dev/toltec --title "Update KOReader to $version" --body "[Release notes](https://github.com/koreader/koreader/releases/tag/v$version)" --label packages --web

[raisjn]

looks great!

[Eeems]

Would it be possible to run this in github actions on a schedule?

[matteodelabre]

Related to #59

[matteodelabre]

As you can see from the issue I linked above, this has been on our backlog for a while! So, thanks a lot for reviving this. I’ll try to abstract some of the ideas of your script to see how we could apply it to other packages. From your script, we can see that there are 3 key steps on which we can work:

version=$(curl -so - https://ota.koreader.rocks/koreader-remarkable-latest-stable.zsync | awk -F '[v.]' 'NR==2 {print $2"."$3}')

1. Fetching the latest upstream version. From what I understand, here you’re using the endpoint that’s used by KOReader itself to fetch auto-updates. I don’t expect most packages to have such a feed available, but we could also watch Git tags, GitHub releases, or RSS feeds provided by the maintainers. In the linked issue, I mentioned nvchecker which can work with a wide variety of possible sources for new version information.

There is also a missing step in your script: comparing the latest upstream version with the version of our package, to decide whether there is actually a new version available. This may be trivial if we kept exactly the same version format as upstream in our pkgver. We should reuse our version parser to extract the upstream component of the package version.

wget -q --show-progress "https://build.koreader.rocks/download/stable/v$version/koreader-remarkable-v$version.zip"
time_stamp=$(date -u -r koreader-remarkable-v$version.zip +%Y-%m-%dT%H:%M:%SZ)
checksum=$(sha256sum koreader-remarkable-v$version.zip | awk '{print $1}')

2. Fetching updated source checksums. We can generalize this to simply iterate over the sources=() array of the package recipe. But what if the source package is corrupted during this wget call? A wrong checksum would then get recorded in the recipe. Can we protect against this?

sed -i -e "s/pkgver=.*/pkgver=$version-1/" -e "s/timestamp=.*/timestamp=$time_stamp/" -e "s/[a-z0-9]\{64\}/$checksum/" ./package/koreader/package
git diff --unified=0
git add .
git commit -m "Update KOReader to $version"

3. Updating the recipe fields. Well… sed works. Not sure if we can do any better here.

[gbyl]

to see how we could apply it to other packages

That is a much more ambitious task! One of the reasons I picked KOReader is because it is a rather non-contentious package. No other packages depend on it. Worst case scenario we push a bad release and KOReader crashes, users go back to xochitl or their launcher of choice. I would be more aprehensive about automating the updates for key packages such as the framebuffer, display, etc.

There is also a missing step in your script: comparing the latest upstream version with the version of our package, to decide whether there is actually a new version available.

Good catch! I actually had that step, but removed it. I learn about KOReader updates when I use their OTA updater, which then prompts me to run my script for Toltec every month or so. I do agree that it needs to be present if this is to be scheduled by maintainers:

[[ $version = $(curl -so - https://raw.githubusercontent.com/toltec-dev/toltec/testing/package/koreader/package | grep "pkgver=" | awk -F '[=-]' '{print $2}') ]] && echo 'Toltec is already up to date' && exit

[Eeems]

I think it would be worth implementing a scheduled action just for KOReader first, and then look at making it more generic later. I don't think any of us maintainers currently have time to do this, but we'd be happy to accept a PR to implement this @gbyl if you'd be willing to create one.

[Eeems]

@gbyl would you be willing to open a PR to add this?

[gbyl]

I can try. I assumed only a few people with special repo permissions would be able to implement this. Since I am not familiar with Github Actions, it may take me a while to figure it out though.

[Eeems]

@gbyl did you ever have a chance to look into this?

[gbyl]

Unfortunately, I could never find the time to look into this. I ultimately settled for pinning the version and managing future updates directly from KOReader. Do you think it would be best to close this issue and take it off the Toltec backlog?

[Eeems]

I think it's still worth having on the backlog. I do want to make sure you are aware that doing what you are doing isn't supported by toltec and could have unintended side effects. I would much prefer that you opened PRs to update KOReader so that we can get it through to stable faster. Ideally, we only lag behind by at most a week, depending on when during the week the PR gets merged into testing.