search

AppraiseDev / Appraise

Appraise code used as part of WMT21 human evaluation campaign
BSD 3-Clause "New" or "Revised" License
22 stars 12 forks source link

No results exported for custom campaign? #108

Closed st-vincent1 closed 1 year ago

st-vincent1 commented 1 year ago

Continuing from #107 , I can successfully build multiple campaigns, but when I make some annotations (either through the web app or command line), and then run python manage.py ExportSystemScoresToCSV [campaign name], nothing gets exported (unlike in the Example/ campaigns where the annotations made get exported to a CSV). Unfortunately, there is no error message either so it is impossible to know what is going on under the hood.

To reproduce:

  1. Clone and install repo from scratch
  2. Download bad.tar.gz and unpack in Appraise/
  3. Make the campaign: python manage.py StartNewCampaign BadExample/manifest.json --batches-json BadExample/batches.json --csv-output BadExample/output.csv

Make a few annotations: for i in range {1..15}; do python manage.py MakeAnnotation [userid]:[password] Direct 70

Try exporting: python manage.py ExportSystemScoresToCSV badexample1

For me, this runs, but no results are printed. In fact, I also ran a loop to annotate all samples in the batch thinking maybe the user needs to complete the tasks first, but still nothing.

By the way, is there a debug mode one could use to address these in the future?

snukky commented 1 year ago

I can reproduce the issue. Will check and let you know.

snukky commented 1 year ago

It's because itemType: in your batches.json includes unsupported value. Supported values are: TGT (translation output), REF (reference), CHK (repetition?), and BAD (so-called "bad" reference). The last three types are used for quality control, so you probably want to choose TGT. Use targetID: to mark which system is a specific item for.

That's again lack of the documentation. Sorry about that. Let me know if you encounter any other issues or have more questions.

st-vincent1 commented 1 year ago

Worked!