Auto generate manifests for newer versions

[felipecrs]

Disclaimer

I tried to find an issue like this in both repositories, but I didn't.

Description of the new feature/enhancement

To support some kind of CI that, based on previously set logic, can auto-generate manifests for newer versions of the packages.

Proposed technical implementation details

A very good example is the Chocolatey Automatic Package Updater Module: https://github.com/majkinetor/au

Based on custom rules, such as scripts for finding newer versions, URLs for downloading them, a CI can run continuously based on a scheduled trigger, and if it finds a newer version, it try to build a new manifest automatically, test it, and submit a PR (or push directly).

So then, we don't need to manually create manifests for every new version of an existing package.

[superusercode]

Scoop also has their own implementation of this: https://github.com/lukesampson/scoop/wiki/App-Manifest-Autoupdate So does chocolatey: https://github.com/majkinetor/au and https://docs.chocolatey.org/en-us/create/automatic-packages

[felipecrs]

If we settle on an implementation archtecture I think the community would be excited to help. I also think this is something to put on the v1 backlog.

I was thinking of something like:

Every package (which has its own folder inside the Publisher) can have a file called update.yml. This file can be like

template:
  Id: GitHub.cli
  Name: GitHub CLI
  AppMoniker: gh
  InstallerType: MSI
updaters:
  - arch: x64
     uses: updater-github-releases
     with: 
       repository: 'cli/cli'
       regex: '*_windows_amd64.msi'
  - arch: x86
     uses: updater-github-releases
     with:
       repository: 'cli/cli'
       regex: '*_windows_386.msi'

And then, updater-github-releases would be a plugin ready to be used for the packages delivered through GitHub Releases.

This enables people to create every kind of plugin, and this way making them reusable without having to share code. People could create plugins which are applicable for a single distributor, but for that case, we can also have some generic plugins, such as updater-html-webscrap:

updaters:
  - arch: x64
     uses: updater-html-webscrap
     with:
       url: https://contoso.com/download
       versionFilter: '.list-group-item:contains(version)'
       downloadUrl: "https://contoso.com/download/contoso_${{ version }}_amd64.msi"

These plugins must return some values, such as version, x64.url, and x64.hash for instance. They can discover hashes by downloading or by any other mechanism, such as another field in the previous example hashFilter: ".list-group-item:contains(${{ version }}).hash".

These plugins could be npm modules for easier community engagement.

[felipecrs]

And then, a new project called winget-updater (could be written in TypeScript for example).

It needs to be a CLI which can be called passing the update.yml or an option for recursively check a folder for update.yml's, and then:

1. Run all the desired plugins which will return the required fields IF a new version is found.
2. IF a new version is found, generate a new manifest using the template area from the update.yml and the values returned by the plugin(s).

[felipecrs]

And finally, to write a CI pipeline (https://github.com/microsoft/winget-pkgs/issues/1515) which would:

1. Run winget-updater on the manifests folder.
2. Submit a PR for each of the newly created manifests

And then the process goes as usual:

1. The Azure Pipelines start testing the PR
2. If passes, Azure Pipelines can check if this PR was opened by the updater bot and automatically merge
3. If not pass, leave the PR open for manual review.

[WtfJoke]

@felipecassiors sounds awesome if you ask me

[dfreilich]

+1 to the idea of adding in some automatic way of registering for updates of our package from Github. I was thinking of adding in a winget package for the Pack CLI (https://github.com/buildpacks/pack/issues/707), and it would be great to minimize the number of steps necessary to keep it updated (so we don't have to do something like this: https://github.com/nodejs/node/pull/34014)

[mimi89999]

I think that the just-install-updater-go (https://github.com/just-install/just-install-updater-go) is very nice. Rules are very simple to write and understand and they work for the vast majority of apps in the registry (https://github.com/just-install/just-install-updater-go/blob/master/jiup/rules/rules.go).

[denelon]

We have started the work for this feature. The first instance of this automation is related to packages that are distributed through a "vanity URL". If the installer changes, the SHA256 becomes invalid. We run a recurring scan on all manifests. If the package has changed, we will generate a new manifest and run the validation. Assuming all of that passes, the new manifest is merged, and the new version is available.

[ghost]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 7 days. It will be closed if no further activity occurs within 7 days of this comment.

[ghost]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 7 days. It will be closed if no further activity occurs within 7 days of this comment.

[mimi89999]

Please don't close this.

[denelon]

The message from the bot wasn't correct. The issue will not be auto closed unless it has the "Needs-author-feedback" and hasn't had any activity for 7 days after the "Status: no recent activity" label has been added. This will not be auto closed based on the current bot rules.

[superusercode]

I'm glad that we now have the current auto update solution for packages that have vanity/static urls (especially the ones only distributed via vanity url). I'm looking forward to it being becoming just as capable as scoop's method for autoupdating packages.

[superusercode]

latest vanity urls from sourceforge may need to be avoided, since their end seems to inconsistently pick up latest versions of apps.

i.e. qbittorrent is at version 4.3.5, but using the link with your browser https://sourceforge.net/projects/qbittorrent/files/latest/download will provide version 4.3.4.1 despite the newer version being listed in the files for the project.

[felipecrs]

I think the use of "latest" URL should always be avoided when possible, to keep the manifests installable even when using an old version.

[Jaifroid]

I would call them "permalinks" rather than "vanity URLs". This feature can be useful if the update can't be included in the app's build toolchain, but it could lead to periods where winget install might fail (when the manifest hasn't been updated yet, but the app has). I guess it also prevents any kind of version history, if a user wants to try an earlier version of an app, or if, like with npm, a developer wants to use winget to establish a particular version of an app or tool as a dependency of something else. It doesn't seem like a good idea to encourage this IMHO.

[denelon]

@Jaifroid, we've been using the term "vanity URL" for quite some time. I don't think "permalink" is necessarily better. In most of the cases I've encountered permalinks are just static URLs to a specific web page, blog, etc. I'd think of a permalink to an installer as more of a URL that points to an individual version of an installer that would never change. That's actually what we would prefer to have so we can continue making "earlier" versions of installers available via the Windows Package Manager.

[Jaifroid]

OK, I stand corrected!

[denelon]

@Jaifroid no worries. More opinions and perspectives will help us build a better product.

[denelon]

Update:

We attempt to generate manifests when we encounter a hash failure. These are generally those pointing to a vanity URL. We have also released wingetcreate to support CI/CD scenarios. The next steps we're considering are extending the ".package" file implemented as a part of #100 to support this kind of automation. Our preference would be to have software publishers generate their own manifest in their CI/CD pipelines, but this might be a reasonable approach to "self-serve" updates. I know some users have built RSS feeds and possibly other mechanisms to detect when a package has changed and produce a manifest.

I believe integration with wingetcreate would greatly reduce the complexity of having all of the metadata in a template, but we would still need to identify the various mechanisms used for publishing.

[felipecrs]

Make everyone use wingetcreate would certainly be the best thing.

But there are some packages in which the developers of it will never care about doing so, but still, we would want them up-to-date in WinGet repo.

Another totally different approach for solving the issue would be to advise the community to create their own repositories for detcting and publishing new versions to winget for a given service. For example, I could create a repository like so:

felipecrs/winget-manifest-upgrader

and inside of it I could have a GitHub Action set to run on a schedule (for example, every day), and then write a script to detect if there is a new version for a package, and if so, use wingetcreate to create a newer manifest for it, and then publish to winget repo.

This would bring us the concept of "trusted community maintainers" or something like that. You could set me as a "trusted community maintainer" of one "orphan" package in winget, for example. Then, I would have the privileges for handling its upgrade.

This approach would require almost zero effort from the Winget maintainers side.

Of course, once https://github.com/microsoft/winget-pkgs/issues/100 is done.

[denelon]

Similar Issues: https://github.com/microsoft/winget-pkgs/issues/1184 https://github.com/microsoft/winget-pkgs/issues/1515

[vedantmgoyal9]

@felipecrs I have already tried to use wingetcreate for auto-updating manifests, but there are some things that need to be improved:

• when running wingetcreate update, it should automatically detect the new version, instead of updating the installer URL in the latest available manifest and in case of generating a new version manifest, we need to pass additional parameter --version
• the logic for matching installer architecture and type should be improved because it mostly gives errors:

  Each new installer URL must have a single match to an existing installer based on installer type and architecture. The following installers failed to match an existing installer:
  Multiple matches found for X86 Nullsoft installer detected from the url: https://github.com/IrosTheBeggar/mStream/releases/download/v5.4.3/mStream-Server-Setup-5.4.3.exe

• the number of InstallerUrls should be counted as 1 where there are two same URLs but the different scopes and installer flags. for e.g. in case of Nullsoft Installer where they have two flags:
  1. Scope: user and the flag is /CURRENTUSER
  2. Scope: machine and the flag is /ALLUSERS

I'm sure wingetcreate will be much more powerful in the near future but till then I am using YamlCreate.ps1 to update packages that are shipped through GitHub Releases or have an API kind of thing where we can check periodically if a new version is available and update the manifests.

[felipecrs]

@vedantmgoyal2009 I would suggest you to post these suggestions as enhancement requests in the wingetcreate repo, if not already. I'm sure the maintainers would be glad to discuss it.

[felipecrs]

This amazing project by @vedantmgoyal2009 needs to be mentioned here:

https://github.com/vedantmgoyal2009/winget-pkgs-automation

[denelon]

I've had a couple of threads with @vedantmgoyal2009 on this. I'd like to find a way to incorporate this directly into winget-pkgs to bring the automation side by side with manifests. We have a few other priorities right now, but it's almost exactly as we had thought about building it. I think the main difference is the naming for some of the JSON keys.

It is amazing work!

[russellbanks]

https://github.com/vedantmgoyal2009/winget-releaser

[Baneeishaque]

what is the status of this feature?

[denelon]

This is partially implemented. For packages with "vanity URLs" we automatically check daily and if the hash has changed, we submit a new manifest.

External automation like what has been built by @vedantmgoyal2009 is something we're looking to directly incorporate, but we have other priorities currently.

Ideally, publishers will automate publishing their manifests. We've had several iterations of wingetcreate and other projects have already built GitHub actions leveraging that tool like PowerToys and Oh My Posh.

[sitiom]

Ideally, publishers will automate publishing their manifests. We've had several iterations of wingetcreate and other projects have already built GitHub actions leveraging that tool like PowerToys and Oh My Posh.

See also: Komac and Winget Releaser

[JustMyGithub]

@denelon That is nice, but how do you know wether a URL is "vanity" or not? I assume "vanity" is that there is a reference to a version denoted in the url, but how do you detect that automatically? (Or is vanity URLs something else?!)

Looking forward to see this extended to a URL and RegEx or URL+ XPath for version check of "non-vanity" URLs as suggest in 15656

Nice to see how winget progresses.

[denelon]

"Vanity" is something like "foo.com/download/setup.exe". The URL never changes, but the software at that URL does.

When there is a version in the URL, it's easier to reason about, and in general the software at that URL does not change. These are the ones we prefer as we are able to maintain older versions of software in the repository.

I don't think we want regex in the manifest. We might consider evaluating regular expressions in tooling to help keep things updated in the future. That's some of the forward thinking about how we might detect when newer versions of software are available. This of course assumes semantic versioning so we can search for "(x+1).0.0" or "1.(x+1).0" or "1.0.(x+1)".

[superusercode]

I don't think we want regex in the manifest.

I agree with this, as I recently learned that scoop's method of auto updating is done via a github action running a powershell script that lives in the bucket repository. At best autoupdate regex could live as a separate file within the package's directory but it's not information the client or the user needs to know or act on - just the updater. It's when the regex breaks that someone has to pay attention to it (i.e. site format change, switched source code host, new non-semver version format, etc.) - and it can get complicated pretty quickly despite scoop providing all sorts of variables to enable it.

The way scoop does it currently makes it harder for the average user to contribute the programs they use since autoupdate and checkver are outright required for manifests, but that in turn has kept their repos up to date even in periods when the maintainer was absent. I don't think there's a single good answer at the moment but making it easier for people to contribute on their own via interactive tools like komac and the existing automated solutions (wingetreleaser, winget update action, etc.) has a more immediate impact.

[BrandonWanHuanSheng]

Sometimes the setup also having a payload which redirect to the setup location https://download.geogebra.org/installers/6.0/ But wget have ability to do this to load the config file from url in the content but the robots need to be off.. If there is new update available, it will attempt to download newer version, wget64 -e robots=off -r -c https://download.geogebra.org/installers/6.0/ without deleting old file.

[sitiom]

wget64 -e robots=off -r -c download.geogebra.org/installers/6.0

I've tried this, and I see that it downloads every file in the directory, so I'm not sure how that helps.

Anyways, the solution for that case is to parse the version from https://download.geogebra.org/installers/6.0/version.txt and fetch the installer URL from there. See https://github.com/ScoopInstaller/Extras/blob/7b1cbb0b29cc0e3fffbd9e392834b2cdd115a1b4/bucket/geogebra.json#L39-L47

[BrandonWanHuanSheng]

I think so but if you keep the app previously for a little while. Because I like to clone web server. If the same file and same size during the next download. wget will respond The file is fully retrieved, nothing to do until it saw a new url in the location..

[BrandonWanHuanSheng]

But github was hard to find their application. Could be there is a new package request somewhere around here?

[pl4nty]

is this still planned? there are a few package update solutions from the community, and I'd love to contribute if they were centralised/standardised