Is there anyway to train "big data" using transformer?

[zqp2009happy]

It sames that Transformer reads training data into the Memory. So it easily got OOM Error with "big training data" like 10G (about 50 million text pairs). Is there some solution for this problem?

[ZhitingHu]

By "Transformer" do you mean the example code under examples/ or the transformer modules in the library?

The transformer modules are independent of how you manage training data (either in memory or disk), as long as you pass to it a data minibatch each iteration.

The transformer example code does load the whole training data into the memory beforehand (code here). To avoid this, you may want to use Texar data module that loads data sequentially. Here is an example use of the Texar data module.