Why you convert the whole dataset to tensor?

[YufanPaPa]

We conduct the Ubuntu experiment follow your pipeline, but we get "ValueError: Cannot create a tensor proto whose content is larger than 2GB". We found that you convert the whole training data to tensor in load_data.py:

def get_batch_data():

# Load data
X,X_length, Y, sources,targets = load_train_data()
# calc total batch count
num_batch = len(X) // hp.batch_size
# Convert to tensor
X = tf.convert_to_tensor(X, tf.int32)
Y = tf.convert_to_tensor(Y, tf.int32)

We are sure that the Ubuntu dataset is mush larger than 2GB, so we are confused how did you do the Ubuntu experiment?

[zhanghainan]

This is a batch of data, not the whole data. You can print the batch data to see if there is some errors in data_load.py

[Sry2016]

I have the same question...

[Sry2016]

In data_load.py, the code likes what @YufanPaPa said. Can you explain about this? Thank you.

[Sry2016]

This is a batch of data, not the whole data. You can print the batch data to see if there is some errors in data_load.py

But in the code, you have use load_train_data(), which will return the whole processed dataset.