Generate badges for implementations

[Julian]

https://shields.io/endpoint

• [x] supported drafts
• [x] per-draft compliance %
• [ ] overall compliance %
• [ ] keywords-with-failing-tests

[VishrutAggarwal]

Hi,

I would like to work on this issue, can I get some more details regarding the same?

1. What exactly should the badges should show and represent?
2. What do you mean by supported drafts?
3. Which file should I look out for while working on resolving this issue?
4. Where should the badges appear?
5. Is there any particular format that the badges should be in?

[Julian]

Hi @VishrutAggarwal thanks for the offer! This'd be super helpful to do, and apologies for the issue description not being very good.

Here's some detail:

The main goal is to produce badges usable via that shields.io link. There are a few different kinds of badges we'd want to generate for every implementation -- and the way you do so essentially is by producing a JSON file (as part of Bowtie's continuous integration) and then pointing shields.io at the URL for this JSON.

Specifically as an example of each:

Supported drafts should be a simple badge that indicates which versions of JSON Schema an implementation supports. As an example, if you run bowtie info -i clojure-json-schema you'll see that the Clojure jsonschema implementation says:

⊙  bowtie info -i clojure-json-schema                                                                                                                                                                                                        julian@Postmac
name: "json-schema"
language: "clojure"
homepage: "https://github.com/luposlip/json-schema"
issues: "https://github.com/luposlip/json-schema/issues"
dialects: [
  "http://json-schema.org/draft-07/schema#",
  "http://json-schema.org/draft-06/schema#",
  "http://json-schema.org/draft-04/schema#"

i.e. it supports draft 7, 6 and 4.

So we want to generate a badge that looks like:

(which I manually generated from this link: https://img.shields.io/badge/JSON%20Schema%20Versions-4%2C%206%2C%207-Green but we would want to generate dynamically for every implementation in the way I mentioned above).

I'll explain the rest of the badges:

• per draft compliance % should be a badge which for each implementation summarizes the results of https://bowtie-json-schema.github.io/bowtie/draft7 (or each draft correspondingly) -- for example, on that draft 7 page, the VSCode JSON Schema implementation fails 106 out of 1176 tests (which is 91% passing) -- so we'd want to generate a badge saying:

which the maintainer of that implementation could put in their README or whatever. And that badge should automatically be updated (in the GH Actions CI job) whenever that implementation passes or fails additional tests.

The overall compliance % badge should basically take into account all versions supported by the implementation -- so if it supports 4, 6, and 7, we'd show a combined number of how many tests it fails across all versions.

And a keywords-with-failing-tests badge would show which keywords an implementation has failing tests for -- for instance, the VSCode implementation looks like it fails tests for the $ref, unevaluatedProperties, unevaluatedItems and a few other keywords, so we'd generate a badge listing those keywords it has failing tests for.

So the main file you'd touch would likely be adding a bowtie badges command to bowtie/_cli.py, and then calling that command from a GitHub action (probably a new one, though you may also be able to do it in the existing test suite report workflow) in https://github.com/bowtie-json-schema/bowtie/tree/main/.github/workflows.

I'd definitely say we don't need to do all of the above for this issue! If you even manage to get one kind of badge dynamically generating (and available for use with https://shields.io/endpoint) that'd be great! You'll need to follow the format that shields.io says we need to generate.

And as for:

Where should the badges appear?

Adding a link to the report https://bowtie-json-schema.github.io/bowtie/ which allows someone to copy-paste a link to the badge would be nice, but isn't 100% necessary either, I think if you just document where someone (an implementer) can go to get a link to their badge that's also OK as a start.

Hope that helps, feel free to ask further if need be!

[VishrutAggarwal]

Thanks!

Since I'm new to this repo, it might take me some time to go through the code. Are there any resources that might help me with this issue? The code in file bowtie/_cli.py is a little overwhelming to look at right now.

[Julian]

It's not directly related, but it might be useful to go through the harness tutorial to get a sense for what Bowtie does, and then to try and follow through the code how it does what it does.

[VishrutAggarwal]

Hi,

These are what I've implemented for now:

1. Created an implementation for badges.
2. Calculated the Passing Percentage for the corresponding draft.

I'm unable to get the draft version for the specific run. I was trying to use the dialect attribute but I was not able to get the draft name/version. Could you point me in the right way so I could work further on resolving this issue?

[Julian]

Which badge are you working on? The supported versions badge? Or the pass percentage one? The former is probably easier to implement, so I'd probably start there, what you need is similar to what the bowtie info command retrieves -- but let me know if you're attempting the other one, if so I can give guidance for that one too!

[VishrutAggarwal]

I currently working on the Pass percentage one, I will start working on the supported versions badge right after I successfully implement at least one badge for now.

[Julian]

Got it -- then you should be able to get the dialect information for a run from RunInfo.dialect -- e.g. if you call _report.from_input(some_report_file)["run_info"].dialect that's a URI which represents the dialect you ran. We probably would want to add a reverse mapping that maps these URIs to some human-readable names for badges (e.g. Draft 4, Draft 2020-12, etc.).

Does that help?

[VishrutAggarwal]

Alright, I tried to implement your suggestion in this way.

Here, I called the summary using the same input and it works flawlessly, but it caused a number of traceback calls for run_info:

Any idea where I went wrong here and what should I focus on to get the desired behavior?

P.S. I have yet to change the variable names so please don't mind them for now.

[Julian]

_report.from_input will consume input, so you can't call it twice on an iterable -- you can split that line into:

report = _report.from_input(input)
summary = report["summary"]
dialect = report["run_info"].dialect

[VishrutAggarwal]

Okay it's working now, thank you for your help!

I'll be working on next steps now.

[VishrutAggarwal]

Hi,

Where can I find all the draft names for reverse mapping? I was trying to look for the names and I came across something. On calling report["run_info"].dialect I get draft-07 in the return value but when I go to here, I get the draft name as draft7

1. I wanted to know if there are any conventions for naming the draft names that I get from dialect? If so, please state those so I could using string manipulation directly on dialect. If not, could you please point me to where I could find the list of all the draft names that would come up in dialect?

2. Also I was having problems in implementing output file here, even though it has been used previously in the exact same manner.

Previously used in this implementation:

The implementation tried by me:

The errors and traceback calls that shows up:

Any idea on how I should have used the commands so it works correctly? Also is there any way I could determine the issue from the traceback calls itself?

3. Finally, I was looking at how to use the endpoint at shields.io/endpoint and on this site, I found that I have to put the JSON file on some site which can host the raw content of the JSON file and then add that site URL to the endpoint so that shields.io can generate the badge. I wish to know if there is any other way to resolve this instead and if you could provide me with either the right resource or just point me in the right direction.

[Julian]

I wanted to know if there are any conventions for naming the draft names that I get from dialect? If so, please state those so I could using string manipulation directly on dialect. If not, could you please point me to where I could find the list of all the draft names that would come up in dialect?

This is the list of "human readable" names for versions -- check the "common name" column there.

Any idea on how I should have used the commands so it works correctly?

You have def badges(input), but you need def badges(input, output) as it will be passed to your function -- the traceback is trying to tell you that (it's passing output to your function but you didn't say it takes an output argument). It's also slightly easier if you paste code or a link to your branch than posting screenshots.

and then add that site URL to the endpoint so that shields.io can generate the badge.

Right! Exactly -- the point is we should be able to generate this JSON file within GItHub actions and then point shields at that page -- I suspect it hopefully should be as easy as simply outputting the JSON file into the GitHub pages bundle which we already generate -- you can probably either try this on your fork of the repository (enable the report workflow and run it, which should generate a site on your username) or you could set up a simple test repo with your JSON file that you've generated and see whether it's hosted properly.

[santhosh-tekuri]

relevant community discussion https://github.com/json-schema-org/community/discussions/118

[VishrutAggarwal]

Thanks for the previous suggestions, I have successfully generated the JSON file and reverse mapped most of the draft names.

I'm facing a lot of issues working on this.

you can probably either try this on your fork of the repository (enable the report workflow and run it, which should generate a site on your username) or you could set up a simple test repo with your JSON file that you've generated and see whether it's hosted properly.

I've been looking into this and I am having problems with understanding how to enable the report workflow, should it not run on its own after testing is done? I am new to working with Github actions and don't understand a lot of how it functions. I'm still reading the documentations and going through the tutorials to understand but I might still need a little guidance in that area.

we should be able to generate this JSON file within GItHub actions and then point shields at that page -- I suspect it hopefully should be as easy as simply outputting the JSON file into the GitHub pages bundle which we already generate

Since shields.io requires a hosted raw JSON file which it can read while generating the badge wherever the link is pasted, I thought of a lot of things that make me doubtful if it would really work that way. If I implement a way to host the raw JSON file temporarily, the badge will show an error once the hosted JSON file gets deleted.

I might be misunderstanding the requirements here, it's majorly because I haven't had much experience with GitHub actions. If you could help me out to clear out something I may have misunderstood or simply am clueless about, it would help me with planning my next move.

Alternatively, I found a way around all of this by just simply using https://img.shields.io/badge/{dialect_name}-{passing_percentage}-brightgreen as a template and outputting this link so that the implementer can copy and paste the link to use the badge. Having said that, I have minimal experience with working with dynamic creation of badges using shields.io. I might be still misunderstanding the requirements.

[Julian]

I've been looking into this and I am having problems with understanding how to enable the report workflow, should it not run on its own after testing is done?

It might be easier / help you to try going to https://pages.github.com/ and creating a simple unrelated GitHub pages site (this is a bit more about pages than actions).

If you create one and stick some HTML in it, and then stick a JSON file next to the HTML and publish that, can you get the HTML to read the JSON file? If so, we can likely do a similar thing here. If not, we need to do something else.

Alternatively, I found a way around all of this by just simply using https://img.shields.io/badge/{dialect_name}-{passing_percentage}-brightgreen as a template and outputting this link so that the implementer can copy and paste the link to use the badge.

The point of doing this is so that as new runs of the suite happen or new tests are added or the implementation gets worked on, the number will change -- so having a static link defeats the purpose essentially, it'd mean the implementer has to continually manually update the badge, rather than deploying a badge which is always up to date with how good the implementation is.

Keep asking if you have trouble! But perhaps the GitHub pages link will get you unstuck.

[VishrutAggarwal]

Hi,

So I've worked on fetching a JSON file to an HTML file and I have successfully done so.

This further led me to experiment with the badge implementation and now if I run the badge command along with the test suite command first to generate the JSON file and then run the report command with the same test suite command to generate the report, I can show the badge with the results of the current test run on the UI.

I've attached an example of the same.

I hope this satisfies the requirement. Please share your thoughts on the same.

Furthermore, I will be working on combining both the commands to report only, so that it can be done in a single command.

Edit: I was pondering about the requirements and I think I have an inkling of what should be done.

Instead of changing the badge link, we just need to change the JSON data, this will help in updating the badge data every time the JSON file is updated without updating the badge manually every time. We'll just add the raw link of the generated JSON file to the endpoint without adding it to the .gitignore file.

[Julian]

Instead of changing the badge link, we just need to change the JSON data, this will help in updating the badge data every time the JSON file is updated without updating the badge manually every time.

Right -- if we can do this as part of the regular run of bowtie suite that happens in GitHub actions, and then publish a permanent link which updates each day, then we should be able to take that link and use it with the shields endpoint to get dynamically changing results.

Sounds like you may have understood the requirements, feel free to put up a PR whenever you think there's something worth looking at!

[VishrutAggarwal]

if we can do this as part of the regular run of bowtie suite that happens in GitHub actions, and then publish a permanent link which updates each day, then we should be able to take that link and use it with the shields endpoint to get dynamically changing results.

I think it'll be a little hard to determine a permanent link without it actually generating the file once. For now I have made an educated guess about the generated JSON file which will be added as a separate file in the repo on basis of which the badge will be generated dynamically using shields endpoint.

I have also added it to the template file so that it can be viewed when the report is generated.

I'll out up a PR and will work on the necessary changes as you suggest them. I will also give and explanation regarding what I did and why I did so if the need arises.

[Julian]

I think it'll be a little hard to determine a permanent link without it actually generating the file once.

I'm not sure what you mean I'm afraid. If we generate a file called badges.json as part of the GitHub Actions run and include it here, that would correspond to the URL https://bowtie-json-schema.github.io/bowtie/badges.json and is what we'd use with shields.io.

generated JSON file which will be added as a separate file in the repo on basis of which the badge will be generated dynamically using shields endpoint.

We definitely don't want to be committing the file to the repo, it changes dynamically. We just need to publish it at that URL via GitHub Pages.

I'll out up a PR and will work on the necessary changes as you suggest them. I will also give and explanation regarding what I did and why I did so if the need arises.

Perfect! Looking forward to it.

[VishrutAggarwal]

I'm not sure what you mean I'm afraid. If we generate a file called badges.json as part of the GitHub Actions run and include it here, that would correspond to the URL https://bowtie-json-schema.github.io/bowtie/badges.json and is what we'd use with shields.io.

Okay, I'm a little unfamiliar with GitHub Actions still so I was looking for ways around that, but I think I should get to it. This might help a lot in understanding where I need to make changes.

We definitely don't want to be committing the file to the repo, it changes dynamically. We just need to publish it at that URL via GitHub Pages.

I understand where I lack now and where I made a mistake, I will get to working on it but I'll need to watch a few tutorials on how GitHub Actions work and which actions would be relevant to this issue. It would help a lot if you could suggest some resources for the same.

For now, this is job I've come up with to add in the workflow file

      - name: Finish setup for badge implementation
        run: |
          touch _site/badge.json

I have yet to figure out how to generate the JSON file using workflow. But I'm a little afraid to make changes directly in the YAML file with no way to test the changes. If you could give me insight on how to do that, it would help a lot.

Though working on this would take me a little while. Can I start working on another Issue in parallel? One that I might be a little more familiar with?

[Julian]

Have you done what I suggested here: https://github.com/bowtie-json-schema/bowtie/issues/55#issuecomment-1473831122 -- it should make things lots easier to see on what you'd touch and how.

The connection between the two is that right now Bowtie's GitHub Pages site is built by Bowtie's GItHub Actions workflow -- once you understand the former (GitHub Pages), the latter should be lots easier, though if you've done it already and still are having difficulty let me know.

Can I start working on another Issue in parallel? One that I might be a little more familiar with?

I wouldn't advise it personally. Working on two things at once is a good way to make lots slower progress on both in my experience, but if you're stuck we can obviously put this issue aside and work fully on some other one, which is totally fine.

[Julian]

For now, this is job I've come up with to add in the workflow file

That line will create an empty badges file. To create a "real" one, if you modify bowtie report to take a --badges <filename> option, then you should be able to modify this one line to pass your new option and create the files (though it occurs to me we may have to combine those files, but once you get one working that part should be straightforward to help with).

But the idea is you should be confident that generating the badges JSON file works when you run it yourself on your computer, and once it does, if you modify that line I linked you can see whether the same thing happens when GitHub actions runs the command using your code.

[VishrutAggarwal]

Have you done what I suggested here: https://github.com/bowtie-json-schema/bowtie/issues/55#issuecomment-1473831122 -- it should make things lots easier to see on what you'd touch and how.

I did and I still did not find what I was looking for. I am currently doubtful of what the structure of the build file actually looks like.

I wouldn't advise it personally. Working on two things at once is a good way to make lots slower progress on both in my experience

Noted, I guess I am just anxious because we're nearing the deadline for the GSoC proposal and I didn't have any merged contribution to show for it. I apologize for any inconvenience this might have caused, I will work on this issue with a more open mindset.

To create a "real" one, if you modify bowtie report to take a --badges option, then you should be able to modify this one line to pass your new option and create the files (though it occurs to me we may have to combine those files, but once you get one working that part should be straightforward to help with).

I think I do not understand how to go about this and do not understand most of the working, mainly due to lack of skill. I will look into how I might be able to accomplish this.

As of now, this is what I've though of:

DONE: For the JSON file to be generated, I will need the report/summary to be generated first so I can get the required data for the badge. The bowtie report command (my edited version) does create a JSON file with the required data and creates it in the same directory as bowtie-report.html.

TODO: Generate the JSON file and copy the contents to _site/badge.json which will then be hosted on the site and can give the raw link for shields endpoint.

OBSTACLE: I do not know where the generated JSON file would be located at so I could copy the contents or if I would even need to copy the contents in case the file is generated at the right location.

If I could have some guidance on how to proceed with this, it will shorten the time I need to properly resolve this issue.

P.S. I looked at my PR and realized my code is not up to date with the latest commits which made it unmergeable and led to failing the checks. Any idea on how I should go about that?

[Julian]

Let's start by getting to the point where your PR can generate a badges JSON file as part of running bowtie suite -- is that the case already? If so I can review it and give you more pointers.

But I want to be able to run:

bowtie suite -i lua-jsonschema https://github.com/json-schema-org/JSON-Schema-Test-Suite/tree/main/tests/draft4/ | bowtie report --badges badges.json

and get two files in the current directory, the bowtie-report.html file we already generate and then a second file called badges.json which should be in the format specified by shields.io and contain badges for all the implementations which ran the tests specified.

Is that already the case? If so I'll try to run your code and show how to modify the GitHub Action workflow!

[Julian]

P.S. I looked at my PR and realized my code is not up to date with the latest commits which made it unmergeable and led to failing the checks. Any idea on how I should go about that?

You can merge the main branch with yours (or rebase, but if you don't know how to do so, use merge). Let me know if you need more specific instructions!

[VishrutAggarwal]

Is that already the case? If so I'll try to run your code and show how to modify the GitHub Action workflow!

Yes! That has already been implemented, just that you do not need to put --badges badges.json running bowtie report automatically generates badge.json in the same directory as bowtie-report.html file

[Julian]

just that you do not need to put --badges badges.json running bowtie report automatically generates badge.json in the same directory as bowtie-report.html file

OK I'll give a shot at running what you have then, but let's make it so it only happens when you include the command line argument -- someone running bowtie report locally probably does not care about generating badges, that will be something useful only when we run in CI / in GitHub Actions, so it should require passing a command line argument to "enable" it.

[VishrutAggarwal]

let's make it so it only happens when you include the command line argument

Okay, I'll look into resolving that. Is there anything I should do before working on that? or anything I should keep in mind as I implement that functionality?

[Julian]

It should only be a 2 or 3 line change, so I'd say if it seems to get more complicated than that feel free to ask.

[VishrutAggarwal]

Hi, I apologize for the delay in update, I was unable to implement --badges badges.json as an option.

This is what I've tried so far:

1. Made _badges as a separate function and call it using a feature switch (PROBLEM: It did not receive it as a file).
2. Used callback parameter in click.option (PROBLEM: I could not make it send required parameters)
3. Used if...else for a quick fix. (PROBLEM: It refused to take --badges as a custom option)

I currently cannot think of a new way to go about it, still I am working on it. If you could point me in the right direction, it will be of great help.

P.S.: I was unwell for the past few days and hence the late replies, but I am better now and will definitely be more active.

[Julian]

If for whatever reason we can't get the shields.io implementation working, I've just had https://github.com/emibcn/badge-action come across my timeline. It seems likely it can be used to do this as well.

[VishrutAggarwal]

Hi @Julian,

If for whatever reason we can't get the shields.io implementation working, I've just had https://github.com/emibcn/badge-action come across my timeline. It seems likely it can be used to do this as well.

Will look into this after implementing the four badges first. Though, I think once all the badges are generated once, the URL for the generated badges will stay constant and can be used for real-time updated badges using shields.io.

Currently, I'm trying to implement keywords-with-failing-tests badge. I came across some doubts:

1. I have no idea on how to get the keywords.
2. What should the badge look like? Can you give an example with a hypothetical draft with 2-3 failing keywords?

If working on overall compliance is preferred before this, I can work on that too but I am not sure how to test if the code generates overall compliance or just the single draft's compliance

UPDATE: I have created the overall compliance badge code, I'll be creating a PR for that in a short while.

[Julian]

I don't think we need to look into that library, Shields.io seems to be working well.

Will send you some info on keywords badge in a bit. Thanks for your efforts!

[VishrutAggarwal]

Okay, let me know regarding this, I might stay offline for a few days to clear out my exams. I will probably look into this in between but I might not be able to put up a PR (special emphasis on might).

But yeah, I can work on it a little in between, it actually helps me with making good use of my time while I'm distracted :bowtie:😓

[Julian]

So it looks like we actually don't pass the keyword information down yet to the report, so that'll need doing first unfortunately (and is slightly related to #27)... Until we do that the information needed for the badge actually isn't available yet :/

Doing that might be slightly more involved @VishrutAggarwal -- if you're looking for something a bit more self contained, maybe another thing you can help with is #264 which I partially added yesterday? Specifically, I added Bowtie to the GitHub Actions Marketplace maybe you can try setting up a small example repository containing 3 files: a JSON Schema, an instance (some data valid under the schema) and then a GitHub workflow which uses that Bowtie action (you can find an example in Bowtie's own workflows) and then once you see that working in your example repository, adding a short bit of documentation to Bowtie's docs? Does that sound like something you might be interested in until #27 gets done?

[VishrutAggarwal]

Okay, that sounds interesting, I'd like to work on it. Though I will need to familiarize myself with what needs to be done, it is still kind of an unfamiliar territory for me.

Can you please assign me that issue? I'd like to avoid what happened last time I took some time working on something I was unfamiliar with.

[Julian]

I have honestly never used the assign feature on GitHub, but sure, if you'd like -- you it seems need to at least comment on the issue for you to be able to be assigned.

[VishrutAggarwal]

Yes, on it. Thank you!