[Feature Request] Badges for ML models

[ashishpawar517]

As this is Github action we can get badges like build passing badges . But It would be great if we would have Custom CML badges that prints the model name with accuracy , or highest testing score etc.

Let me know if what you guys think about this ?

[elleobrien]

@pawarashish564 we've been discussing how to do this! Here's what we have in mind, let us know what you think:

• A badge that looks something like this:

  

• Color. Some GH badges use colors as cues (greener badge --> better on some metric), but since there are many model metrics and some are minimized rather than maximized, we would default to blue or another neutral color.

• One badge per metric. You might have several metrics in your experiment, and each badge will represent one metric. Badges should be stackable so you can have several.

• What do badges represent? In a project with several branches, you might have multiple versions of the model each with different scores. Assuming you will eventually merge your best model into the main/master branch of your project, the badge can represent metrics on main (instead of the best of any branch). The badge will be updated on a merge event.

• User interface. Because ML projects vary so much it might be worthwhile to provide a CLI method users can set in a workflow. We'd have to think about implementation, but since there are many kinds of model metrics that are handled in many ways (some are tracked by DVC, some are stored in text files and not tracked beyond that) we could use a syntax like cml-publish to afford customizability:

cml-badge-publish <metrics file in .csv/.json format> <desired metric, if multiple are present>

[ashishpawar517]

Yes that what's I'm looking for . Can you tell me How you did this ? In detail. Is it simple prototype or working demo for all? Thank you so much.

[elleobrien]

We haven't made it yet! We'll get working on it and let you know when it's ready. I made the sample accuracy badge using shields.io :)

[0x2b3bfa0]

(Bikeshedding)

The dynamic-badges action uses an interesting approach, but might not be appropriate in some scenarios due to the public nature of GitHub Gists. Would extending the assets backend be in consonance with the current practices? https://github.com/iterative/cml/blob/cdde17c8eb212c56b343fe9433c055ad3a05f873/src/utils.js#L67

[elleobrien]

I believe we had decided not to use the assets backend because the badge needs to be at a static address so it can be referenced in the README.md perpetually, and the assets service uses content-addressable storage.

[0x2b3bfa0]

The next most plausible option I can think of would be spinning up a serverless function tied to a cheap schemaless database. For the badge use case, I think that it would be enough to keep only the most recent result for each branch on a repository.

[casperdcl]

Got a solution for you (feel free to feedback)... have a look at https://github.com/iterative/dvc.org/issues/234#issuecomment-848243511

[ashishpawar517]

I agree, very interesting

[DavidGOrtega]

@casperdcl @pawarashish564 Is this issue right now fixed?

[ashishpawar517]

Yes