Getting first and last hidden outputs for BiLSTM with variable-length sequences

[ccsasuke]

Hi,

I have a zero-masked bidirectional RNN for mini-batched data with variable lengths. I want to extract the last hidden output of the forward pass and the first hidden output of the backward pass as the representation of the sequence.

Since each sequence in the batch has different lengths, I can think of two possible ways to implement this: i) Prepare two versions of the data, one with zeros padded on the left while the other with zeros padded on the right. The forward pass uses the left-padded data and use SelectTable(-1). Similarly, the backward pass uses the right-padded data and use SelectTable(1).

ii) Alternatively, I can implement a custom MaskZeroSelectTable(-1) to select the last hidden output that is not 0.

I guess this is not an uncommon usage, but I could not find an existing solution. Am I missing anything? If there is no over-the-shelf implementation, what is the best way to implement this?

[aron-bordin]

Hi @ccsasuke, I'm facing the same issue. Have you found any good solution ?

[ccsasuke]

Hi @aron-bordin, it seems this didn't catch the attention of the devs.. I switched to some other framework for this function. Maybe you can try the workaround i) I mentioned?