jackboyla / GLiREL

Generalist and Lightweight Model for Relation Extraction (Extract any relationship types from text)
90 stars 6 forks source link

Add auxiliary confidence head #3

Open jackboyla opened 4 months ago

chrishokamp commented 4 months ago

Goal: At some point during training, we wish to introduce an additional confidence head, which will output model confidence that a given prediction is correct.

current loss computation https://github.com/jackboyla/GLiREL/blob/main/glirel/model.py#L242-L259 backprop: https://github.com/jackboyla/GLiREL/blob/main/glirel/model.py#L242-L259

chrishokamp commented 4 months ago

Note multiple heads can be trained for sub-objectives, for example just for the prompt representation https://github.com/jackboyla/GLiREL/blob/main/glirel/model.py#L64-L70 and encoding a prior probability that the trained model will succeed at predicting a given zero-shot relation

q-chrishokamp commented 4 months ago

Flair subtoken pooling magic https://github.com/flairNLP/flair/blob/a852bff2b8c3cab3dcf7bd19fb1c528225816bf6/flair/embeddings/transformer.py#L233-L239