Use `tf.data` API to build flexible and high performance data pipeline

NTMC-Community / MatchZoo

Facilitating the design, comparison and sharing of deep text matching models.

Apache License 2.0

3.82k stars 898 forks source link

Open luozhouyang opened 4 years ago

luozhouyang commented 4 years ago

[x] I checked to make sure that this is not a duplicate issue
[x] I'm submitting the request to the correct repository (for model requests, see here)

We need a more flexible and powerful data pipeline when training on very large corpus.

Use tf.data API to build the high performance and flexible data pipeline.

faneshion commented 3 years ago

Do you mean the tf.data can not handle large-scale dataset? Did you try the MatchZoo-py version?