Viewer shows outdated cache after renaming a repo and creating a new one with the old name

[albertvillanova]

Reported by @lewtun (internal link: https://huggingface.slack.com/archives/C02EMARJ65P/p1719818961944059):

If I rename a dataset via the UI from D -> D' and then create a new dataset with the same name D, I seem to get a copy instead of an empty dataset Indeed it was the dataset viewer showing a cached result - the git history is clean and there's no files in the new dataset repo

[julien-c]

is is possible to also list the solution we discussed in that thread? (moving from repo name to _id)

To make discussion a bit more efficient

[lhoestq]

in particular:

_id field in hf.co/api/datasets - guaranteed immutable for a given repo

Some first thoughts:

• we can add this field to jobs created when we receive a webhook (or for any job creation)
• add this info in the cache entries
• update to a _id-centric logic what is currently based on the repo name:
  • it concerns S3 assets and cached assets, as well as parquet metadata on NFS
  • since we store the full locations in the cache entries, this can surely be done without immediate migration of existing files locations

Then regarding the API:

• the API calls from the hub could use the _id in addition to the repo name to make sure it gets the right data (even if there is outdated data in the cache somehow)

I also considered using _id everywhere as the source of truth, but I anticipate it will just move the problem elsewhere to the place we will cache the _id <-> repo name mapping (repo name is always needed to read/write to repos and also for the dataset-viewer API)

[albertvillanova]

Thanks for the complementary information, @lhoestq.

So, basically we would need a complete refactoring of all the logic to identify repositories and you also think that this would just move the problem elsewhere... :thinking:

I am wondering if instead we could effectively face the real underlying problem, that is, properly handle the repository renaming event, even if a new repository with the old name is created.