Open pngwn opened 1 month ago
Very interesting, would love to be more involved in these discussions
I think the solution is to write guides/tutorials on this (could have a dedicated section for this) as opposed to directly integrating anything into core gradio. The HF Datasaver abstractions have proved brittle and users always want to do something custom anyways ("I actually want each chatbot response to be a separate row in the dataset").
I agree that the current abstractions aren't very useful but I don't agree with just delegating to users via tutorials, will edit the OP with some context / thoughts.
I agree with @pngwn, from a user persona that recently wanted to implement this. It required 3 manual steps to actively and continuously capture input and output from Gradio apps while it could be really powerful to make this more integrated out of the box, like the Rest-API availability. Especially for iterating on your data and models.
As from a PR I made recently. https://github.com/davidberenstein1957/gradio/pull/2
from transformers import pipeline
import gradio as gr
pipe = pipeline("text-classification", model="mrm8488/bert-tiny-finetuned-sms-spam-detection")
demo = gr.Interface.from_pipeline(pipe)
demo.sync_with_hub(
repo_id="davidberenstein1957/bert-tiny-finetuned-sms-spam-detection",
every=1
)
demo.launch()
or
from transformers import pipeline
import gradio as gr
pipe = pipeline("text-classification", model="mrm8488/bert-tiny-finetuned-sms-spam-detection")
demo = gr.Interface.from_pipeline(pipe)
demo.sync_with_argilla(
dataset="davidberenstein1957/bert-tiny-finetuned-sms-spam-detection",
every=1
)
At Argilla we also used to have these monitors that functions as pipeline wrappers.
from argilla import monitor
from transformers import pipeline
pipe = pipeline("text-classification", model="mrm8488/bert-tiny-finetuned-sms-spam-detection")
monitored_pipe = monitor(model=pipeline, dataset="my_dataset", log_interval=1, sample_rate=0.3)
I also created 2 example spaces based on the ChatInterface
Thanks @davidberenstein1957, the sync_with_hub
api is very nice, but I'm wondering whether it is necessary given that we already have support for a custom FlaggingCallback. Instead of creating another abstraction, I think the right thing to do would be to create an example using FlaggingCallback that syncs data to the Hub / Argilla, and then including that as an example for users in our docs. (We should also ensure that flagging works as expected -- iirc you had mentioned that flagging does not work when submitting data from the client)
Yess @abidlabs, it did not seem to log data when sending requests through the client.
I think we could focus on the docs or usage examples to start (1 minute video "collect human feedback with gradio and sync that to the hub") but as I understood from @petrov826 that the team was going to work on Gradio 5.0, I assumed it would be a good excuse to review the API design given that Pete mentioned there were mixed feelings about the accessibility of the current implementation.
Not saying we should use my PR, but consider revisit the FlaggingCallback. Perhaps renaming to something like DataCollectionCallback would the discussed intended use more explicit. Also, perhaps changing some argument defaults. I was more thinking in such a direction.
I feel like the last piece is a guide on synching to the hub and also implementing a custom flag method for a chatbot or something like that
Adding this note from https://github.com/gradio-app/gradio/pull/9166 so I don't forget
Argilla integration, dataset integration etc.
detail to follow.