Extend homebrew badge to support 3rd-party taps

[jasonkarns]

:clipboard: Description

As homebrew recommends more and more formulae to live in 3rd party taps, it would be necessary for these tapped formulae to have badges pointing to the 3rd-party tap.

Homebrew itself already has a format for installing from a 3rd party tap directly:

brew install owner/tap/formula would point to formula under: github.com/owner/homebrew-tap

As such, my request would be that the existing homebrew badge just accept formula names that include the owner/tap/ prefix.

[chris48s]

At the moment, the data from the homebrew and homebrew cask badges come from the API on https://formulae.brew.sh For example, if you request the badge https://img.shields.io/homebrew/v/python we're populating that badge by calling https://formulae.brew.sh/api/formula/python.json Would the data for this come from the same place? Can you work through an example of this?

[jasonkarns]

Ah, gotcha. Quickly scanning the homebrew service JS, it looks like versions.stable is the only property from that JSON that shields needs?

Homebrew has a command (brew info --json <formula>) which generates that JSON. (sample below generated from a tap formula that I own) The structure is the same as is available through their API. (The API also includes analytics information for the formula—download counts, installs, etc—which wouldn't be available from the locally-run version of the command, but doesn't appear necessary?)

So a hypothetical: if this JSON data were available in the tap itself as a static file, a homebrew-tap badge service could query the tap repo for the file and parse the formula's JSON instead of hitting the brew.sh API.

It would be fairly trivial for a tap author to setup, say, a GitHub action which generated the JSON file for each formula in the tap on pushes to master...

So all that is left would be to agree on a conventional location for the JSON file to live within the tap... perhaps in the same directory as the formula so you'd have formula.rb and formula.json next to each other; perhaps in separate folder that mirrors Formula, or perhaps even in a separate branch? (considering that these taps are git-cloned by users...)

Example JSON generated from a local formula in one of my taps. This would be the same thing potentially generated (and committed) by a GitHub Action:

$ brew info --json Formula/nodenv-aliases.rb | jq

[
  {
    "name": "nodenv-aliases",
    "full_name": "nodenv-aliases",
    "oldname": null,
    "aliases": [],
    "versioned_formulae": [],
    "desc": "Create aliases for nodenv Node versions",
    "homepage": "https://github.com/nodenv/nodenv-aliases",
    "versions": {
      "stable": "2.0.2",
      "devel": null,
      "head": "HEAD",
      "bottle": false
    },
    "urls": {},
    "revision": 0,
    "version_scheme": 0,
    "bottle": {},
    "keg_only": false,
    "bottle_disabled": false,
    "options": [],
    "build_dependencies": [],
    "dependencies": [
      "nodenv"
    ],
    "recommended_dependencies": [],
    "optional_dependencies": [],
    "uses_from_macos": [],
    "requirements": [],
    "conflicts_with": [],
    "caveats": null,
    "installed": [],
    "linked_keg": null,
    "pinned": false,
    "outdated": false
  }
]

[jasonkarns]

I've spiked out a github workflow which generates the necessary json and commits it back into the tap under an Info/ directory. An example of the generated JSON is here: https://github.com/nodenv/homebrew-nodenv/tree/json/Info (the workflow is in the same branch, if you're curious).

So my proposal is for the shields service for homebrew taps:

1. the tap formula badge would accept the formula name using Homebrew's "fully qualified" name: owner/tap/formula
2. derive the github repo for the tap (this is easy because a tap name of foo/bar should exist at github.com/foo/homebrew-bar per Homebrew's naming guidelines)
3. optional: if desired, shields could optionally fallback to using a github repo of github.com/foo/bar if the homebrew-bar form doesn't resolve to an existing repository. this would accommodate taps that do not conform to homebrew's naming structure
4. now the service can make a request to https://raw.githubusercontent.com/{owner}/homebrew-{tap}/master/Info/{formula}.json to get the formula's json info to derive the version, etc.

Example:

https://img.shields.io/homebrew/v/nodenv/nodenv/nodenv-aliases

would fetch the JSON at https://raw.githubusercontent.com/nodenv/homebrew-nodenv/master/Info/nodenv-aliases.json

[paulmelnikow]

Would it be feasible to extract the version number from the .rb file? It would be nice if badge users didn’t have to add a build step.

[jasonkarns]

@paulmelnikow to be clear, the build step would be taken on by tap owners, not all badge users. (though that's probably a distinction without a difference since it's quite unlikely that anyone other than tap owners would use this badge :D )

Would it be feasible to extract the version number from the .rb file?

I'm sure it would be possible; though attempting to extract it without homebrew's own logic would be quite brittle. Firstly, the version is extracted from multiple places within the rb file, depending on the type of formula and the download strategy. Additionally, I'm sure there are all kinds of rules within homebrew that govern edge cases dealing with formula versions (ie, what happens if the formula itself is revisioned despite the tool itself not having a version bump...). So I don't think any kind of regex or vanilla ruby parsing is a great choice.

That said, the build step in this case is just calling brew info --json <formula>. I'm unfamiliar with the architecture of shields.io to know whether there is any possibility of doing this JSON extraction directly within the shields service. homebrew does support linux, so it's technically feasible to have homebrew installed on the service box in order to run this command.

Also, not that it really matters, but the build step I spiked out in my tap is virtually identical to what homebrew core does for its "API". The entirety of the formula.brew.sh site (including api routes) is completely statically generated (jekyll). So the .json routes currently being queried for core formula are just returning the JSON generated by the site's build step. So if this feature puts a burden on tap maintainers to generate their own JSON, it's not any more of a burden than homebrew-core is doing for its own tap.

[paulmelnikow]

Yea, I understand. To use the badge you'd need to get the tap owner to set this up, so I meant user as a tap owner who decides to put the badge on the project, not someone who's consuming the tap.

I'm a little reluctant to settle for a solution in Shields that requires the tap owner to set up an action in order to compute the data, because it makes me think relatively few people will do it. We'll be maintaining half of a custom system, which only tap owners can opt into.

There are two ways that already exist to render badges for custom stuff in Shields, which could be combined with the action.

1. Point the Dynamic badge at the JSON file you're generating, providing a JSONPath query to fetch the version.

2. In the GitHub Action, transform the generated JSON file into a JSON file in our Endpoint badge format:

{
  "schemaVersion": 1,
  "label": "nodenv/nodenv-aliases",
  "message": "v2.0.2",
  "color": "blue"
}

These would require a little more work when setting up the Action or badge, but would mean Shields doesn't have to maintain/support/document how to set up the GH Action workflow.

Alternatively, perhaps we can find a way to support this in Shields that works automatically in most cases, and makes a slight and hopefully not onerous imposition on the tap owners.

For example, from looking at your formulae, I imagine we could try to parse the url line automatically. For cases where that doesn't work, the tap owner could be required to drop in a VERSION = ... line at the top, which could look something like this:

VERSION = "2.0.2"

class NodenvAliases < Formula
  ...
  url "https://github.com/nodenv/nodenv-aliases/archive/v#{VERSION}tar.gz"

If we detect that VERSION = line, we'd give it priority.

This way if someone wants to create a set of badges for the third party taps they use, they can, probably without having to bug all the tap owners.

In the past we've had some services in the past which fetch a bunch of data and/or run external processes, such as a badge for Bower, and I think we should not go back in that direction on the main Shields server. If running brew info --json <formula> is the best way to generate the JSON file, it would be a good use for a microservice which does only that, which could then be used as a Shields endpoint.

I'd really like to support this – on the Shields server if we can – though I'd strongly prefer that it just work on most taps, without having to ask the tap owner to set up additional tooling.

[chris48s]

I was just in the process of typing a response to this, but I think Paul has broadly covered it. Essentially, in lieu of a centralised resource, I think in order for us to use this approach (or something similar), it would need to be an established standard or at least a very widely adopted community convention. I guess the percentage of homebrew taps with this json file in their repo is pretty close to 0% today. i.e: its a very bad assumption to assume this exists.

One of the options mentioned above is the dynamic json badge. Just to work that through, if you just want to do this for your own projects and generate a badge based on an arbitrary json file in your github repo, you can do that like this:

https://img.shields.io/badge/dynamic/json.svg?url=https://raw.githubusercontent.com/nodenv/homebrew-nodenv/json/Info/nodenv-aliases.json&query=$.versions.stable&label=version

[paulmelnikow]

I guess the percentage of homebrew taps with this json file in their repo is pretty close to 0% today. i.e: its a very bad assumption to assume this exists.

That's really well put.

[jasonkarns]

This is all fantastic! I agree with almost everything that has been said thus far.

I've been mostly convinced that just regex-ish parsing the ruby file is probably the right thing in these conditions; since that would probably cover 80% of 3rd party taps, automatically, without any work by the tap owner.

I'm extremely hesitant to recommend the VERSION extraction, though. Homebrew is extremely particular about the format of formula. The rubocop (ruby linter) styleguide is extremely pedantic, and it only gets moreso over time. So even in in the likely case that we could contrive an easily-parseable extraction, it's possible that a future change to their styleguide would put it in violation. But I'd wager that the standard github-url format would successfully cover 80% of 3rd party formulae; and there's probably a second-tier format that covers the next 15%.

And if the above all end in failure, the custom json endpoint is a viable fallback for my own tap(s).

I'll look into spiking a PR to do the github-url parsing soon.

Are you comfortable with extending the existing homebrew service such that this new logic automatically kicks in when a user provides a fully-qualified (owner/tap/formula) formula name?

And secondly, is there already some utility for handling the case where a badge's primary "input" contains slashes?

I guess the percentage of homebrew taps with this json file in their repo is pretty close to 0% today. i.e: its a very bad assumption to assume this exists.

For sure. I wasn't suggesting that this Info/*.json convention already exists, but rather that we create this convention. (Since the majority of the people using this badge would be tap owners who wish to include the badge on their tap page would be in a position to actually make their tap conform to the convention.)

[chris48s]

Just to check - is homebrew intrinsically tied to github. i.e: is it impossible to have a homebrew package where the repo lives in gitlab/bitbucket/something else ?

[paulmelnikow]

I think the answer is yes, it's specific to GitHub: https://docs.brew.sh/Taps

Are you comfortable with extending the existing homebrew service such that this new logic automatically kicks in when a user provides a fully-qualified (owner/tap/formula) formula name?

I think making it homebrew/tap/v/:user/:repo/:formula would be better. It'll be simpler to implement too. If these are always coming from GitHub, the tap formula should extend GithubAuthV3Service so it can leverage the token pool.

[jasonkarns]

is it impossible to have a homebrew package where the repo lives in gitlab/bitbucket/something else

Strictly speaking, no it's not impossible. Homebrew itself will work okay as long as the tap is a git repository. For instance, it's fairly common for companies to host private taps on their own instance of GitHub Enterprise (which is still github, of course, but to your point, is not github.com).

However, if a tap is not on github, the tap is already in for a world of pain. brew tap foo/bar will not work, nor will brew install owner/tap/formula. In both cases, hosting taps outside github.com means that tap users must use full git-cloneable URLs wherever the tap name is used.

I would imagine to accept alternative hosts, we could have a named parameter that defaults to github but could be provided as gitlab, bitbucket, etc? We could have pre-baked URL templates that correspond to those git hosts.

[prantlf]

I am able to generate a JSON file for the custom badge using a Makefile according to @chris48s' example, when I update the formulae Thank you!

https://img.shields.io/badge/dynamic/json.svg?url=https://raw.githubusercontent.com/prantlf/homebrew-tap/master/Info/saz-tools.json&query=$.versions.stable&label=homebrew