zerospeech / benchmarks

A command line tool that helps use the "Zero Ressource Challenge" benchmarks
https://zerospeech.com/toolbox/
GNU General Public License v3.0
8 stars 2 forks source link

Add ProsAudit benchmark #15

Closed nhamilakis closed 1 year ago

nhamilakis commented 1 year ago
nhamilakis commented 1 year ago

With this a2faf4db936c6f4a3b62adb0c4065dfac06b9977 commit, i added the basic integration layout for the new benchmark.

When task is more clear, add :

nhamilakis commented 1 year ago

With this edf89260609d94a81f24866d4a26727af5672696 commit added the continuation of a2faf4db936c6f4a3b62adb0c4065dfac06b9977

TODO:

https://github.com/zerospeech/benchmarks/blob/edf89260609d94a81f24866d4a26727af5672696/zerospeech/benchmarks/tasks/lm/prosody.py#L34-L45

- [ ] Leaderboard entry (once scores have been evaluated)

nhamilakis commented 1 year ago

3f51bc122f62b909e9143109e99d5204969fdd4b fixes mismatch on prosody scoring by_pair.

Syntactic benchmark had transcription and by_pair scoring was done using transcriptions as id. For prosody benchmark transcription column is null, so by_pair scoring needs to contain id column to be identifiable.

nhamilakis commented 1 year ago

d58a500f014915a9884387870ac46f45b9d71bef added leaderboard schema

nhamilakis commented 1 year ago

d63e0585c236e92581ecf6b49ce3e9ae4bc79576 added leadereboard generation