search

zerospeech / benchmarks

A command line tool that helps use the "Zero Ressource Challenge" benchmarks
https://zerospeech.com/toolbox/
GNU General Public License v3.0
8 stars 2 forks source link

Incorporating Zero Resource Speech Challenge into Dynamic SUPERB #38

Open juice500ml opened 4 weeks ago

juice500ml commented 4 weeks ago

Thank you a lot for maintaining this wonderful challenge! I felt the challenge is necessary especially for the recent general speech models.

To further leverage the existing challenge, I thought it would also be nice to incorporate ZRC to Dynamic SUPERB, so that ZRC would be used by even more researchers.

As per the description from the official repo, https://github.com/dynamic-superb/dynamic-superb Dynamic-SUPERB is a dynamic and collaborative benchmark aimed at building universal speech models ... to perform multiple tasks in a zero-shot fashion. It provides a platform for researchers and developers to evaluate and compare different models in various speech-processing tasks. In other words, it will be a benchmark for general-purpose speech models, similar to the ideas of NLP benchmarks such as GLUE, SuperGLUE, BIGBench, etc.

I felt there was a considerable amount of intersection in the motivation, and I believe that, by incorporating some of the ZRC's tasks in Dynamic SUPERB, we will have a much more accurate benchmark. I will be handling other technical issues. such as implementation, etc., but I wanted to ask zerospeech organizers before going further with this direction. I added the outline of the potential plans on the Dynamic SUPERB issue: https://github.com/dynamic-superb/dynamic-superb/issues/140

nhamilakis commented 3 weeks ago

Hello, i see no issue with the inclusion of the zerospeech tasks into your benchmarks, the tasks are released under an open source licence. We would appreciate it if the correct people are cited (see zerospeech.com for details).

I really like your project and will share it with my team, i think some of them will be interested in it. The zerospeech benchmarks were designed as tasks in a challenge, this module is an attempt at converting it into a benchmark, but yes i see a lot of similarities with your project. Will be happy to help if any questions arise, but i do not have much time to dedicate to this project, for the time being.

juice500ml commented 3 weeks ago

@nhamilakis, thanks a lot for your reply! On the implementation-side, me and my colleagues (@eunjung31, @kalvinchang) will start working on this. I'll keep this issue open for now, for possible questions.