fix when using input seq lens

[vieting]

So far, we had no batch-time-major case, but there I think we need the following fix when ignoring entries that are beyond the sequence lengths.