search

yikangshen / Ordered-Neurons

Code for the paper "Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks"
https://arxiv.org/pdf/1810.09536.pdf
BSD 3-Clause "New" or "Revised" License
577 stars 101 forks source link

Processing of variable length sequences in a batch #5

Closed GYM13 closed 5 years ago

GYM13 commented 5 years ago

Hello, I just started learning the language model.I am very interested in your method after reading your paper.But after I read the paper carefully, I have a question and I would like to ask you.But after I read the paper carefully, I have a question and I would like to ask you for advice.In the paper, you directly divide the words in the corpus into equal-length batches.But now every sentence in a batch is different in length, how should I handle it?I queried the handling of the official pytorch documentation（nn.utils.rnn.pad_packed_sequence）.But don't know if this method is right for your code.Can you please give me some advice? Thanks

shawntan commented 5 years ago

nn.utils.rnn.pad_packed_sequence this method wouldn't work with the way this model is implemented.

Depending on what your needs are, you might have to either:

  1. Mask the irrelevant hidden states (if you are doing something like an attention mechanism), or,
  2. Extract the final hiddenstate of the sequence, you can look at our logical inference code for how we did that.