search

gradio-app / gradio

Build and share delightful machine learning apps, all in Python. 🌟 Star to support our work!
http://www.gradio.app
Apache License 2.0
31.98k stars 2.38k forks source link

Better data capture story? #8791

Open pngwn opened 1 month ago

pngwn commented 1 month ago

Argilla integration, dataset integration etc.

detail to follow.

abidlabs commented 1 month ago

Very interesting, would love to be more involved in these discussions

freddyaboulton commented 1 month ago

I think the solution is to write guides/tutorials on this (could have a dedicated section for this) as opposed to directly integrating anything into core gradio. The HF Datasaver abstractions have proved brittle and users always want to do something custom anyways ("I actually want each chatbot response to be a separate row in the dataset").

pngwn commented 1 month ago

I agree that the current abstractions aren't very useful but I don't agree with just delegating to users via tutorials, will edit the OP with some context / thoughts.

davidberenstein1957 commented 1 month ago

My thoughts

I agree with @pngwn, from a user persona that recently wanted to implement this. It required 3 manual steps to actively and continuously capture input and output from Gradio apps while it could be really powerful to make this more integrated out of the box, like the Rest-API availability. Especially for iterating on your data and models.

Recent PR

As from a PR I made recently. https://github.com/davidberenstein1957/gradio/pull/2

from transformers import pipeline

import gradio as gr

pipe = pipeline("text-classification", model="mrm8488/bert-tiny-finetuned-sms-spam-detection")
demo = gr.Interface.from_pipeline(pipe)

demo.sync_with_hub(
    repo_id="davidberenstein1957/bert-tiny-finetuned-sms-spam-detection",
    every=1
)
demo.launch()

or 

from transformers import pipeline

import gradio as gr

pipe = pipeline("text-classification", model="mrm8488/bert-tiny-finetuned-sms-spam-detection")
demo = gr.Interface.from_pipeline(pipe)

demo.sync_with_argilla(
    dataset="davidberenstein1957/bert-tiny-finetuned-sms-spam-detection",
    every=1
)

Monitoring models according to old Argilla API

At Argilla we also used to have these monitors that functions as pipeline wrappers.

from argilla import monitor
from transformers import pipeline

pipe = pipeline("text-classification", model="mrm8488/bert-tiny-finetuned-sms-spam-detection")

monitored_pipe = monitor(model=pipeline, dataset="my_dataset", log_interval=1, sample_rate=0.3)

LLM data collectors for ChatInterface

I also created 2 example spaces based on the ChatInterface

The flow I envision

  1. Start a project
  2. Quickly demo and share (with Gradio along with REST-API)
  3. Use captured data to iterate (in Argilla, also related to the)
  4. Fine-tune (with TRL/AutoTrain/Transformers)
  5. Evaluate, redeploy and continue at step 2
abidlabs commented 1 month ago

Thanks @davidberenstein1957, the sync_with_hub api is very nice, but I'm wondering whether it is necessary given that we already have support for a custom FlaggingCallback. Instead of creating another abstraction, I think the right thing to do would be to create an example using FlaggingCallback that syncs data to the Hub / Argilla, and then including that as an example for users in our docs. (We should also ensure that flagging works as expected -- iirc you had mentioned that flagging does not work when submitting data from the client)

davidberenstein1957 commented 1 month ago

Yess @abidlabs, it did not seem to log data when sending requests through the client.

I think we could focus on the docs or usage examples to start (1 minute video "collect human feedback with gradio and sync that to the hub") but as I understood from @petrov826 that the team was going to work on Gradio 5.0, I assumed it would be a good excuse to review the API design given that Pete mentioned there were mixed feelings about the accessibility of the current implementation.

Not saying we should use my PR, but consider revisit the FlaggingCallback. Perhaps renaming to something like DataCollectionCallback would the discussed intended use more explicit. Also, perhaps changing some argument defaults. I was more thinking in such a direction.

abidlabs commented 1 week ago

I feel like the last piece is a guide on synching to the hub and also implementing a custom flag method for a chatbot or something like that

Adding this note from https://github.com/gradio-app/gradio/pull/9166 so I don't forget