search

microsoft / JARVIS

JARVIS, a system to connect LLMs with ML community. Paper: https://arxiv.org/pdf/2303.17580.pdf
MIT License
23.55k stars 1.96k forks source link

Evaluation Dataset mentioned in Hugging GPT paper is not available #208

Open ssdasgupta opened 1 year ago

ssdasgupta commented 1 year ago

As mentioned in the paper - "Furthermore, we also invite some expert annotators to label task planning for some complex requests (46 examples) as a high-quality human annotated dataset. We also plan to further improve the quality and quantity of this dataset to better help us to evaluate the LLM capability in planning, which leaves as future work.", are you planning to release the evaluation dataset? Or if it is there already in the repository, could you send me the folder location?

Thanks.

StillKeepTry commented 1 year ago

@ssdasgupta We are currently working with our labeling teams to iteratively improve the quality of this dataset and our legal team to ensure compliance of the dataset release. We will release a work about this dataset in the future. Please be patient.

Belonng commented 4 weeks ago

Hello @StillKeepTry ,

I hope you’re doing well. I wanted to kindly follow up on the status of the evaluation dataset mentioned in the previous discussion. I understand that the team has been working on improving the quality and ensuring legal compliance. Could you please provide any updates on when we might expect the release of this dataset?

This dataset would be extremely valuable for my work, and I’m sure many others in the community are also eagerly awaiting it. Your efforts are greatly appreciated.

Thank you for your time!