search

mlabonne / llm-datasets

High-quality datasets, tools, and concepts for LLM fine-tuning.
1.96k stars 174 forks source link

Add ProjectDev to the Agent section #8

Open minhngh opened 2 months ago

minhngh commented 2 months ago

Dear Mr. mlabonne,

I would like to request the inclusion of the ProjectDev dataset in the Agent section. This dataset is used to evaluate the performance of multi-agent systems, such as ChatDev, MetaGPT, and AgileCoder, in solving software development challenges.

Thank you very much.

mlabonne commented 2 months ago

Hey, is it this dataset https://huggingface.co/datasets/Fsoft-AIC/ProjectDev? If that's the case, sorry but this is too small to include in this repo. I'm mostly interested in large datasets (>10k, preferably >100k).

minhngh commented 2 months ago

Yes, that’s correct. The dataset is pretty small because it requires manual evaluation to assess the capabilities of multi-agent systems like ChatDev and AgileCoder. Tasks within the dataset involve generating software based on user input, with user interfaces that feature complex interactions, such as chains of mouse and keyboard events, and generated programs can vary significantly in terms of code element names and implementations. Thus, it's hard to scale the dataset beyond 10,000 samples.