NLT DF file, feedback , requests and scoring

[rwgdrummer]

Feedback is two parts: (1) The current score. (2) the actual writer identifiers of given UIDs from the DF file.

The score reported will include penalties for each writer id requested.

Details are in the coordination plan.

Each response is a tuple: (boolean, string) where boolean is a the validity claim for the Text Review and String is the actual writer id. +1 if one is correct +2 if both are correct, 0 if none are correct. The DF file has the correct values.

The DF is structured as follows (columns order change, so use the DF columns as indicators) instanceid, text, reported_writer_id, real_writer_id, sentiment, product, novelty_indicator, novel_instance

instanceid = uuid text= '|' quoted text stripped of new lines

used for paragraph separators. '||' is used for pipe. sentiment = number reported_writer_id = id to be sent in request real_writer_id = writer id to be used for scoring product = string product id

Each request is sent a CSV file with the instance id, text, sentiment, product id and reported_writer_id:

id,writer_id,text,sentiment,product_id b3b28ca7-d55e-4ade-ab94-ad59af03e60f,001,|It was great|,4,aliens cff908d5-728b-4aa6-912a-7a2e4690d1d1,002,|It was bad|,1,aliens 203293e9-4132-4a64-81f8-8f8586b701ad,013,|I was horrified|,1,aliens 5fbfcf64-a414-4cd6-a1ab-9ecd86d66807,004,|I was elated|,5,aliens 8a6805b2-8ce3-41f3-8eca-7a567c87e3a1,094,|My dog was sleeping”|,3,aliens e8a65311-142d-4ebb-badc-1640c9c3ffe5,202,|Educational at best|,3,aliens 2eaeed87-d169-4df0-8892-98ef84c6a01f,002,|^&()(!#@@|,1,aliens aa21b0cf-a325-49de-8754-b7ad88e69d1d,013,|Lost an hour of my life|,1,aliens

Each response TA2 Agent provides the two-tuple correctness indicator (1 is correct, 0 is not incorrect) and writer label.

Example : b3b28ca7-d55e-4ade-ab94-ad59af03e60f,1,001 cff908d5-728b-4aa6-912a-7a2e4690d1d1,1,002 203293e9-4132-4a64-81f8-8f8586b701ad,1,013 5fbfcf64-a414-4cd6-a1ab-9ecd86d66807,0,001 8a6805b2-8ce3-41f3-8eca-7a567c87e3a1,0,076

Feedback request for writer is the example ids: b3b28ca7-d55e-4ade-ab94-ad59af03e60f

The response is the real writer id: b3b28ca7-d55e-4ade-ab94-ad59af03e60f,2023

NOTE: This is the Reported Score for FEEDBACK. This is NOTE the evaluation score. The evaluation score is an asymptotic score relative to prior batches (the trend). That equation is pending to be placed in the evaluation metrics.

[rwgdrummer]

NEed to support UTF-16 CSV file encoding for DF file and Request OUPUT!