XL-Lexeme-ordinal with multiple thresholds

[Garrafao]

According to @Tuo-ZHANG 's thesis you can choose some thresholds on cosine similarity of the model to predict judgments of 1, 2, 3, 4 pretty well.

@Tuo-ZHANG Could you maybe suggest some good sets of thresholds for the model which you derived on different data sets (languages) and different data set distributions?

@shafqatvirk Could you maybe implement an "ordinal" besides the "binary" option to the model allowing us to provide 3 thresholds as input parameters?

@AinaIanemahy We could implement for the beginning just one ordinal annotator with a fixed threshold set. But, we should decide whether we want to make this a parameter in the future. Do we? Could we may be provide default parameters but still let users choose? That would maybe not confuse them too much, as I think many users cannot make informed choices and may be overwhelmed by too many options.

[shafqatvirk]

I have teaching heavy couple of weeks, it is only after that I can implement this.

[AinaIanemahy]

We would have to store the information on the threshold somewhere in the task table. This would require changes to said table. Not a huge problem, but requires some attention. We could then implement the default thresholds on the DURel side, as well. The information on the thresholds would be passed on as a part of the task information dictionary ('thresholds': [1,2,3]). This would mean that we would have to add a parameter to x1_lexeme_annotate.py that takes an array of threshold. This array would have to be passed on to the judgment making process.

[Garrafao]

@shafqatvirk Before you close this, could you add integration tests for this annotator? See also #22.

[AinaIanemahy]

I think the integration tests are now there so this can be closed. fb6997f

[shafqatvirk]

We don't have integration tests for Random yet. Best Regards

On Thu, Feb 29, 2024 at 2:16 PM AinaIanemahy @.***> wrote:

I think the integration tests are now there so this can be closed. fb6997f https://github.com/Garrafao/durel_system_annotators/commit/fb6997f4446e631cec2876d86c86f664d310504d

— Reply to this email directly, view it on GitHub https://github.com/Garrafao/durel_system_annotators/issues/14#issuecomment-1971116923, or unsubscribe https://github.com/notifications/unsubscribe-auth/AATLDK6Q7IRP6E5TCFSRD2DYV4UZRAVCNFSM6AAAAAA6WPVPGOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZRGEYTMOJSGM . You are receiving this because you were mentioned.Message ID: @.***>

-- Shafqat Mumtaz Virk