search

zihangdai / xlnet

XLNet: Generalized Autoregressive Pretraining for Language Understanding
Apache License 2.0
6.18k stars 1.18k forks source link

Question about `b_target` #79

Open graykode opened 5 years ago

graykode commented 5 years ago

https://github.com/zihangdai/xlnet/blob/master/data_utils.py#L316

I wonder why this is b_begin: b_end + 1, not b_begin+1: b_end + 1.

Also, what is mean?
https://github.com/zihangdai/xlnet/blob/master/data_utils.py#L634

# new target: [next token] for LM and [curr token] (self) for PLM
    new_targets = torch.cat([inputs[0: 1], targets[: -1]], dim=0)

As always, thank you!!

kimiyoung commented 5 years ago

b_begin was designed to be the target that corresponds to SEP, although it was not used in the end because special symbols have no target.

graykode commented 5 years ago

@kimiyoung I understand! So, you are intend to assert tgt.shape[0] == seq_len as vaild Thanks!

image