[CODE IMPROVEMENT] Import dataset from Hugging Face

[pascal-pfeiffer]

🔧 Proposed code refactoring

Allow simple import of dataset from Hugging Face

from datasets import load_dataset
import pandas as pd

Load your dataset
dataset = load_dataset('fka/awesome-chatgpt-prompts')  # replace with the name of your desired dataset

 Convert to pandas DataFrame
df = pd.DataFrame(dataset['train'])

 Save DataFrame to csv
df.to_csv('my_dataset.csv', index=False)

Motivation

Raised by Maher on Discord (https://discord.com/channels/1097462770674438174/1100718594809147402/1136414136495001680)

[pritthakkar]

@pascal-pfeiffer I would like to work on this issue, can you please give me more details that what exact changes I have to make? Can you please add hactoberfest label to this issue?

[pascal-pfeiffer]

Thank you @pritthakkar , this would be great!

What we had in mind here is to add another data connector to the "Import dataset" page here:

code wise, this would need to be added to https://github.com/h2oai/h2o-llmstudio/blob/main/llm_studio/app_utils/sections/dataset.py#L75 ff

The import page could borrow the style of the kaggle import page:

I'd suggest 2 fields:

• Huggingface token (pre-filled from settings page/env var)
• Name of the dataset on Huggingface ("fka/awesome-chatgpt-prompts") in the example above

Things to consider:

• How to deal with datasets that consist of train/val/test or only parts of it.
• Name of the output file and dataset