Closed hubert98 closed 3 years ago
Hi @hubert98, making multi token labels more easy to use is something that I've been wanting to do for quite some while, but I haven't found the time to do so yet. Unfortunately, I'll be on vacation starting tomorrow until January 6th, but I'll try to provide you with an example as soon as I'm back. In the meantime, you may take a look at task_helpers.py
(in particular, RecordTaskHelper
might be of interest) but this is different from your use case in that the set of labels is different for each input example, and the current implementation is not really optimized with regards to readability.
Hi @timoschick - happy new year and hope you had a great time. I am a bit lost with the RecordTaskHelper - how would the labels (e.g. "very good") go in the respective PVP ?
Maybe you can give a short example for dummies (me)?
Hi @hubert98 and @timoschick, I am also interested in this idea of using multiple tokens to represent a label. I think it was mentioned in the home page, in this paper from the same authors.
Please let me know if I have understood the idea correctly (based on section 3.1, equation 4). I believe the core idea is multiplying together the probability distributions of the [MASK] token if there are multiple [MASK] tokens.
Let x be the original text, P(x) be the cloze question, q = M(v(y)|x) be the logits generated by the MLM, M. If we use the same labels and verbalizations that you have provided:
P(x) = 'some text. In summary it was [MASK]'. P^2(x) = 'some text. In summary it was [MASK_1][MASK_2]'. (subscript 2 means there are 2 mask tokens) q(good|x) = M(good|P^2(x)) q(very good|x) = M(very|P^2(x)) \times M(good|P^2(x))
Hi @hubert98 and @TZeng20,
I have now added a way of using multiple masks that does not require you to manually define a custom TaskHelper
.
You can find all relevant information here: https://github.com/timoschick/pet#pet-with-multiple-masks
I didn't have the time to test this feature thorougly, so please let me know if anything doesn't work as expected or if you have further questions by raising a new issue.
I am not sure if this is already found in the code for the identity function solution in your recent paper:
When I want to use multi-token labels - for example: 1) very good 2) good 3) ok 4) bad 5 very bad
very good / very bad are multi-token.
Can you give me a short (for dummys) idea how to implement train that?