search

cfedermann / Appraise

Appraise evaluation system for manual evaluation of machine translation output
http://www.appraise.cf/
BSD 3-Clause "New" or "Revised" License
74 stars 37 forks source link

Improve usability of inter-annotator agreement computation in status view #22

Open cfedermann opened 11 years ago

cfedermann commented 11 years ago

Compute inter-annotator agreement (IAA) for the number of annotators or coders (C) which maximises the number of items (I) that have been evaluated by the respective sub set of coders. This avoids showing no IAA scores until all coders have completed the task.

Additionally, there could be checkboxes which can be used to toggle whether a system should be included in IAA computation or not. That way, different scenarios could be tested in a more playful way, from within the status view.

If implemented, the checkbox selection should also be reflected when downloading results data.

cfedermann commented 9 years ago

Will have to clean this up once I get to refactoring eval tasks into parallel modules...