search

SynoCommunity / spksrc

Cross compilation framework to create native packages for the Synology's NAS
https://synocommunity.com
Other
3.03k stars 1.23k forks source link

Automate new upstream version availability detection and processing #2656

Open ymartin59 opened 7 years ago

ymartin59 commented 7 years ago

Here is first specification of my idea. Python sounds me a good choice for implementation.

Expected behavior

How to use

When over, browsing files marked as new or modified in git working copy is straight forward.

With luck and probably for most minor updates, removing log files and review before commit may be the only remaining job

WORK-IN-PROGRESS: https://github.com/SynoCommunity/spksrc-check-update

TODO: Document usage and required Makefile variables so that it keeps on working as expect

GuillaumeSmaha commented 7 years ago

I made a POC to do this, I will share it when I will obtain a good result. Currently, I just checks the link availibility.

@ymartin59 What do you think about try to ping on possible new file by incrementing the version ? It can be lesser specific than parsing html page.

Also parsing Makefile can be complex when there is specific call like boost which use substr (https://github.com/SynoCommunity/spksrc/blob/master/cross/boost/Makefile#L4). Currently, substr is the only special case.

GuillaumeSmaha commented 7 years ago

You can find the POC here : https://gist.github.com/GuillaumeSmaha/5545f6e48b5d3ce35399d6b3727c4a9c#file-check_package_availability-sh

The result on master : 

cross/domoticz -> Not Available: svn://svn.code.sf.net/p/domoticz/code/trunk
cross/iksemel -> Not Available: http://iksemel.googlecode.com/files/iksemel-1.4.tar.gz
cross/imagemagick -> Not Available: http://www.imagemagick.org/download/ImageMagick-6.9.5-10.tar.xz
cross/jappix -> Not Available: http://download.jappix.org/1.0.1/jappix-1.0.1-one.zip
cross/libcap2 -> Not Available: http://ftp.sunet.se/pub/Linux/kernel.org/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.gz
cross/libev -> Not Available: http://dist.schmorp.de/libev/libev-4.22.tar.gz
cross/memcached -> Not Available: http://memcached.googlecode.com/files/memcached-1.4.15.tar.gz
cross/mpd -> Not Available: http://downloads.sourceforge.net/project/musicpd/mpd/0.16.7/mpd-0.16.7.tar.bz2
cross/phpmemcachedadmin -> Not Available: http://phpmemcacheadmin.googlecode.com/files/phpMemcachedAdmin-1.2.2-r262.tar.gz
cross/pyload -> Not Available: https://github.com/pyload/pyload/releases/download/v0.4.9/pyload-src-v0.4.9.zip
cross/serf -> Not Available: https://serf.googlecode.com/files/serf-1.2.0.tar.bz2
cross/squidclamav -> Not Available: http://downloads.sourceforge.net/project/squidclamav/squidclamav/6.10/squidclamav-6.10.tar.gz
cross/sslh -> Not Available: http://www.rutschle.net/tech/sslh-v1.17.tar.gz
cross/zsh -> Not Available: http://www.zsh.org/pub/zsh-5.2.tar.xz
GuillaumeSmaha commented 7 years ago

Another issue to fix : Sometimes few packages like native/mono or cross/rtmpdump doesn't have a SPK_VERS variable but in case of mono, there are SPK_VERS_MAJOR and SPK_VERS_MINOR and for rtmpdump, this is a git repo as source. It will be better to define all time SPK_VERS even if the var is just equal to the HASH for git or to the concatenation of version variables :

PKG_VERS = $(PKG_VERS_MAJOR).$(PKG_VERS_MINOR).$(PKG_VERS_PATCH)
GuillaumeSmaha commented 7 years ago

Here the first result of my script : http://pastebin.com/azZyyn2Z Script : https://gist.github.com/GuillaumeSmaha/5545f6e48b5d3ce35399d6b3727c4a9c#file-search_package_update-sh

ymartin59 commented 7 years ago

@GuillaumeSmaha Great job but from my point of view such code would have been more robust and easier to maintain if written with Python, typically when error handling is concerned.

GuillaumeSmaha commented 7 years ago

@ymartin59 Yes, python is more robust. I planed to stop the script at this point. The aim was to check if it was possible to have good results. And the next dev needs to be more specific.

GuillaumeSmaha commented 7 years ago

I created a repository with the python script which returns the same result as the bash script : https://github.com/GuillaumeSmaha/spksrc-check-update

(svn dependency required the PR https://github.com/dsoprea/PySvn/pull/85 to work)

GuillaumeSmaha commented 7 years ago

Yeah ! First step is finished ! The script can now check the current version or get the new version for all package except these :

cross/dobby : Old github upload API. And the package no more used
cross/domoticz :  Server down
cross/ejabberd :  Server down
cross/gc :  Error to fix : Script can't download page content http://hboehm.info/gc/gc_source/
cross/jappix :  Server down
cross/libcap2 :  Server down
cross/mpd : mpd is not found in the project https://sourceforge.net/projects/musicpd/files/
cross/nzbget-testing : Hard to detect because the package uses `PKG_VERS` and `PKG_REV`
cross/pear : No version in package
cross/pyload : File is not available. Need to update and check patches.
cross/unzip : Error to connect to the ftp server.

I also create the PR #2708 to add a new var DOWNLOAD_PAGE which defines the page with the files list. For example : On http://www.sqlite.org/, there is not a download link to a file, and the list of files is located on page http://www.sqlite.org/download.html

Updated : cross/gc is now fixed using an User-Agent in the HTTP request

Updated 2: cross/nzbget-testing can be solved by removing PKG_REV and adding revision in PKG_VERS See patch nzbget-testing.patch.txt

ymartin59 commented 7 years ago

@GuillaumeSmaha You really did a great job. Do you mind if I request your repository to be hosted by SynoCommunity organization for issues and pull requests ? I really think it makes sense

May I ask you for a small improvement: follow DEPENDS and BUILD_DEPENDS when fetching a dedicated package ? In that case, running with --package=spk/ffmpeg would report all included cross/ and native/ libraries required by the tool ?

GuillaumeSmaha commented 7 years ago

@ymartin59 I miss your message and no time the last months to do it. But come back to continue. For the build part, it could be interesting to check if we could create a PR and use travis (for example) to validate the build

ymartin59 commented 7 years ago

You're right for building. A ticket specifically concerns Travis support: #2990

Safihre commented 6 years ago

@GuillaumeSmaha This looks great, I see there's 2 branches, what does the feature/add_build do?

hgy59 commented 6 years ago

I don't like automatic updates. An automatism for finding broken downloads and to force SPK_VERS updates is fine. But a package update should only be done if it is worth to, and this should be carefully selected. Each package version should be handled like a LTS version (since there is not way to download an older version anymore after update in package center).

For broken downloads I recently found very different reasons:

GuillaumeSmaha commented 6 years ago

The aim of the branch add_build is to update Makefile and build package, but this part is not already done. Currently, I can detect and found new version for package.

GuillaumeSmaha commented 6 years ago

I impress myself when I reviewed the code of add_build branch. I totally forgot the improvement done. Summary:

@ymartin59 I saw your post: Yes, I agree to move it to SynoCommunity organization

noplanman commented 6 years ago

This looks pretty, what's the status?

hgy59 commented 4 years ago

What about implementing this as a bot? Like dependabot that looks for known vulnerabilities in dependent python modules, this could be implemented as a bot, that notifies about packages, for which updated sources are available. The first benefit would be the notification, but as dependabot, it could provide a PR with the update. And for a PR we will have the github build checks too.

And I am dreaming of automated package installation and functional testing with a virtual DSM environment, implemented as github action...

publicarray commented 3 years ago

I toyed with a simpler approach by only supporting GitHub releases. It has a different use case, but I think making updates / contributions easier should be the goal not full automation (package updates can break in unforeseen ways). I like how homebrew has a command that checks for an update and if there is one it makes required changes and prepares a PR, they also have a package audit script but that's for another time. My quick attempt: https://gist.github.com/publicarray/a5ce25b72d1e13252a918ee5c934f19e

rbrownwsws commented 3 years ago

I thought I would have a go at learning GitHub actions so I made a proof-of-concept dependabot-alike for spksrc like you suggested @hgy59.

At the moment it is a bit rough and will try to automatically create PRs so I would not suggest testing it on the real repo until it is more polished.

Current Limitations:

I've tried to write it to be extensible so hopefully adding new sources & version formats will not be too hard.

Action source code: https://github.com/rbrownwsws/spksrc-deps Example repo: https://github.com/rbrownwsws/spksrc-deps-playground

N.B. Needs minor patches to the spksrc framework see: https://github.com/rbrownwsws/spksrc/tree/deps/framework

rbrownwsws commented 3 years ago

I've polished the action a bit over the last couple of days. It is still pretty rough but at least now should not create PRs for entirely broken patches.

Please tell me what you think about it and if you would be interested in me developing it any further.

I've put instructions below for how to run it yourself if you want to play around with the code.

Beware if you do run it with createPullRequests: "true" as of this morning it will create 33 pull requests.

Local testing

Install NodeJS (I've been using 14 LTS)

Create a token for local testing here: https://github.com/settings/tokens Give it repo and workflow scopes if you want to let it create PRs

Make a new clone of spksrc for the action to play with. It will change the repo user.name and user.email config and create lots of branches so you do not want it using your normal working copy. Merge the deps/framework branch from https://github.com/rbrownwsws/spksrc/

If you want to play with Pull Requests create a new bare GitHub repo, set it to be the origin of your local repo and push.

Edit the following environment variables to fit your local setup:

export GITHUB_TOKEN="<secret>"
export GITHUB_WORKSPACE="/mnt/c/Users/rbrow/Documents/workspaces/sync/syno/spksrc-deps-playground"
export GITHUB_REPOSITORY="rbrownwsws/spksrc-deps-playground"
export INPUT_CREATEPATCHES="true"
export INPUT_CREATEPULLREQUESTS="false"
export INPUT_CREATEISSUES="false"

Get deps:

npm install

Run the action locally:

npm run build && npm run start

N.B. If you run this too many times in quick succession you may hit a GitHub API quota and have to wait for it to reset.

This is helpful to clean up all the branches it makes:

for branch in $(git branch | grep "deps/cross\|deps/native"); do git branch -D $branch; done;

GitHub testing

Add this workflow to a fork of spksrc with the deps/framework patches:

name: Check-deps

on:
  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

jobs:
  checkdeps:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2

      # Run the spksrc-deps action
      # N.B. Do not use `master` in a real project. Pin at a tag instead
      - uses: rbrownwsws/spksrc-deps@master
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        with:
          createPatches: "true"
          createPullRequests: "true"