huggingface / dataset-viewer

Backend that powers the dataset viewer on Hugging Face dataset pages through a public API.
https://huggingface.co/docs/dataset-viewer
Apache License 2.0
702 stars 77 forks source link

remove call to the Hub when processing a webhook #2570

Open severo opened 8 months ago

severo commented 8 months ago

Since https://github.com/huggingface/moon-landing/pull/9061 (internal), the webhooks now contain the list of updated refs, which should make the following call redundant:

https://github.com/huggingface/datasets-server/blob/60bcc28d3bcbc27ced08c29966aa332af666d02c/services/api/src/api/routes/webhook.py#L84-L88

We should now look at the updatedRefs field to know if the webhook is about the refs/convert/parquet branch, for example, or if it modified the main branch.

See the example webhook:

{
  "event": {
    "action": "update",
    "scope": "repo.content"
  },
  "repo": {
    "type": "model",
    "name": "coyotte508/test-s3",
    "id": "634440800f69ad8aa61849ec",
    "private": false,
    "url": {
      "web": "http://localhost:5564/coyotte508/test-s3",
      "api": "http://localhost:5564/api/models/coyotte508/test-s3"
    },
    "headSha": "1d8b8303d5a009d34bc95021c10860c29a5e1644",
    "owner": {
      "id": "61d2f90c3c2083e1c08af22d"
    }
  },
  "webhook": {
    "id": "65d9f396bbebdd1f43d027fe",
    "version": 3
  },
  "updatedRefs": [
    {
      "ref": "refs/heads/main",
      "oldSha": "d26fe083101f2a0c881871c926b09b5c2ca6f15b",
      "newSha": "1d8b8303d5a009d34bc95021c10860c29a5e1644"
    },
    {
      "ref": "refs/heads/test33",
      "oldSha": "ba1b55b341646885b83e746fba77b9fd41e29f87",
      "newSha": "f06ec96ac5a8fa90f05500d9fdf14ea720166602"
    }
  ]
}
severo commented 8 months ago

See the docs at https://huggingface.co/docs/hub/webhooks#code-changes