Open Airuibadi opened 5 years ago
This PR provides a Code Example of what you need.
SingleTaskDataset class is a simple abstract of single-task dataset.
It has three parts: data_reader
, Vocabulary
and Counter
Just look at the following two files to understand how to use data IO.
Reader
Dataset builder
If you have a complicated way to build a batch, you can override the SingleTaskDataset.get_batches
method.
Hope the code example can help you solve the problem. ps. A more detailed documentation example is in our plan.
Can't figure out such highly abstract code in your example, some description would help ,at least some brief comment. Please.