[Python] truncation_length does not work well for user defined deserializer

[main76]

CNTK version: 2.3/2.4, Python version: 3.5.2, CPU. Just like what I have mentioned in the title, and there is my unit test code:

d1 = CLinguisticDeserializer(in_scpFile, schema_path, dict(name='clf', shape=in_dim))
d2 = HTKFeatureDeserializer(StreamDefs(htk=StreamDef(scp="archiveInput_chunk.scp", shape=in_dim)))

# mbsrc1 = MinibatchSource([d1], randomize=False, frame_mode=True)
# mbsrc2 = MinibatchSource([d2], randomize=False, frame_mode=True)

mbsrc1 = MinibatchSource([d1], randomize=False, truncation_length=10)
mbsrc2 = MinibatchSource([d2], randomize=False, truncation_length=10)

for i in range(8):
    mb1 = mbsrc1.next_minibatch(640)
    mb2 = mbsrc2.next_minibatch(640)
    clf_data = mb1[mbsrc1.streams.clf].data.asarray()
    htk_data = mb2[mbsrc2.streams.htk].data.asarray()
    print(np.array_equal(clf_data, htk_data))

Test results for frame mode are all true, while for BPTT are not. Then, I launched this in debug mode, and I found that the shape of clf_data was (640, 10, ), did not meet the expectation of (64, 10, ).

>>> repr(mb1[mbsrc1.streams.clf])
'MinibatchData(data=Value([640 x 10 x 617], CPU), samples=640, seqs=640)'
>>> repr(mb2[mbsrc2.streams.htk])
'MinibatchData(data=Value([64 x 10 x 617], CPU), samples=640, seqs=64)'

I wonder if there are ways I can do to correct it.

[ke1337]

Truncated length only works with non-frame mode according to the code here. Besides, 640/64 in your value tensor should be the batch dimension, not the sequence dimension, 10 seems to be the sequence dimension that matches truncation_length. There might be some validation issues though for truncated and frame mode.

[main76]

It was running on non-frame mode, as the default value for frame_mode is false, ref. By the way, happy Spring Festival!