ValueError: could not broadcast input array from shape 2*x into shape x

[anvarganiev]

My dataset's __getitem__ returns image of shape (1, 32, 128) and np.array of shape (1900). I made a writer:

writer = DatasetWriter(write_path, {
            'images': NDArrayField(shape=(1, 32, 128), dtype=np.dtype('float32')),
            'grids': NDArrayField(shape=(1900,), dtype=np.dtype('float32'))
        }, num_workers=8)

And when I call: writer.from_indexed_dataset(train_dataset) there is an error: ValueError: could not broadcast input array from shape (32768,) into shape (16384,) Wrapping image and array into a tuple at __getitem__ doesn't help (from https://github.com/libffcv/ffcv/issues/103#issue-1112940398) But it works if I change NDArrayField(shape=(1, 32, 128), dtype=np.dtype('float32')), to NDArrayField(shape=(2, 32, 128), dtype=np.dtype('float32')), in writer. but of course then it will fail because of wrong input shape (2, 32, 128) in the model. Cannot understand what is wrong here.

[andrewilyas]

It sounds like your dataset is return float64 rather than float32 (hence the writer is getting twice as many bits as it is expecting). Converting the output of getitem to float32 (recommended) or making the dtype of the fields np.dtype("float64") should both resolve this error.

[anvarganiev]

@andrewilyas you solved my problem. Thank you! It is training now, but there was 47 it/sec during casual training, with FFCV it's 53it/sec.