search

huggingface / transformers

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
https://huggingface.co/transformers
Apache License 2.0
132.11k stars 26.32k forks source link

Include better versions of a model when they are available in the model doc pages #19998

Closed sayakpaul closed 1 year ago

sayakpaul commented 1 year ago

@NielsRogge provided a great suggestion.

We could add a banner on top of Swin's docs (and other models where this is applicable), to indicate we have a better model now, SwinV2. The same could be done for DETR, as Conditional DETR and Deformable DETR greatly improve the convergence and AP metrics.

We can start with the following:

And then expand to the other models. Or else, we can start with five such models (feel free to suggest more). When reviewing model PRs to transformers, we would just need to be mindful about this a bit so that we can suggest it accordingly to the contributors.

Cc: @osanseviero @nateraw

sgugger commented 1 year ago

I am very wary about this. If you start telling on the page of a model that have been developed by Org 1 that there is a better model developed by Org 2, they will get mad and we will then have unnecessary conflicts to handle.

For the same reason we stay away from benchmarks between frameworks/hardware, I would stay away from this.

sayakpaul commented 1 year ago

Point noted.

But what if the same org comes up with a better version? But I understand this creates a weird distinction which is not desirable.

Leaving it open for today in case anyone has any inputs.

NielsRogge commented 1 year ago

Yeah it was just a suggestion, I wouldn't have opened an issue for this actually.

I agree that this could become very opinionated (it's very subjective which model is better). We could just do it for papers that come from the same team (Swin => Swinv2), to promote upcoming work. But for models that originate from different teams, this might be hard. That's where "evaluate on the hub" will come into play, where people can see which models perform best on a given task.

sayakpaul commented 1 year ago

Closing it for now. Should there be a need, it can easily be reopened.

osanseviero commented 1 year ago

We could just do it for papers that come from the same team (Swin => Swinv2), to promote upcoming work.

Sure, why not :) for those scenarios I think it's ok to open small PRs updating the corresponding docstrings