Open lkqnaruto opened 2 years ago
Hi @lkqnaruto ,
By interleaved zeros and ones, I meant a mask like [0, 1, 0, 1, 1, 0, 0, 0, 1, ...]
instead of [1, 1, 1, 1, 0, 0, 0]
. Because we are using WordPiece that is subword tokenization, all word continuation tokens (that start with ##
) do not have an associated tag prediction for NER task, otherwise words that are tokenized into 2+ tokens would have multiple predictions.
For instance, suppose we have these tokens:
tokens = ["[CLS]", "Al", "##bert", "Ein", "##stein", ...]
The mask would be
mask = [0, 1, 0, 1, 0, ...]
which is incompatible with the CRF package. So we have index the sequence using the mask and pass only
["Al", "Ein", ...]
to CRF.
The mask is different for each sequence of the batch and have different lengths (sum of 1's), so this masking is not trivial to do without an explicit for loop.
Hi @lkqnaruto ,
By interleaved zeros and ones, I meant a mask like
[0, 1, 0, 1, 1, 0, 0, 0, 1, ...]
instead of[1, 1, 1, 1, 0, 0, 0]
. Because we are using WordPiece that is subword tokenization, all word continuation tokens (that start with##
) do not have an associated tag prediction for NER task, otherwise words that are tokenized into 2+ tokens would have multiple predictions.For instance, suppose we have these tokens:
tokens = ["[CLS]", "Al", "##bert", "Ein", "##stein", ...]
The mask would bemask = [0, 1, 0, 1, 0, ...]
which is incompatible with the CRF package. So we have index the sequence using the mask and pass only["Al", "Ein", ...]
to CRF.The mask is different for each sequence of the batch and have different lengths (sum of 1's), so this masking is not trivial to do without an explicit for loop.
Thank you for the reply, a follow up question: Why we have to loop through each batch before crf? I think crf package can handle batch-wise calculation.
Hi @fabiocapsouza,
I'm experimenting with different ways of subword handling for CRF layer. Why have you chosen to just take first subtoken? Wouldn't some sort of pooling of subword representations work better?
I would greatly appreciate if you could share your thoughts on the matter!
Hi @ViktorooReps , I used the first subtoken because it is the way BERT does it for NER, so it is the simplest way to add CRF on top of it. Yeah, maybe some sort of pooling could be better, even though subword representations are already contextual. It would be a nice experiment.
I checked out the code, where you said :
Can you explain a little bit on that? I'm still confused what you mean here. What does it mean by "interleaved zeros and ones"?
Thank you