Multiple HuggingFace identifiers can point to the same dataset

[PGijsbers]

E.g. both links below resolve to the NaiveDev version of the dataset:

• https://huggingface.co/datasets/aha-org/coco-2014-instance and
• https://huggingface.co/datasets/NaiveDev/coco-2014-instance

I think this is because the user changed their name (aha-org doesn't exist anymore). Yet in AIoD, metadata for each is registered independently, and so they are counted as two datasets. I feel like this is not what we want, and we should update the connector to reflect that. Logging this here, with the intention to discuss this Thursday Nov 7.

• Should we add a new entry with the new name? Delete the old one? Or update the status? Or should we instead update the old entry? How should we detect this happened? It is only observable when trying to visit the old link, so a new index wouldn't immediately trigger any alarms.

[PGijsbers]

The number of duplicate registrations may be relatively low, using the publication date as a proxy:

+--------------------------------+
| count(distinct date_published) |
+--------------------------------+
|                         237955 |
+--------------------------------+
1 row in set (2.12 sec)

mysql> select count(*) from dataset where platform="huggingface";
+----------+
| count(*) |
+----------+
|   250897 |
+----------+
1 row in set (0.24 sec)

We should also check if there are other assumptions that may be incorrect, e.g. can a user also change the name of the dataset? In any case, while we should also challenge the assumption that the publication date remains static throughout the lifetime of the dataset, if we assume this to be true, then it should put an upperbound of erroneous double indexing at ~13k regardless of the source of this duplication.