Improve data handling for training and inference - Githubissues

ybracke / transnormer

A lexical normalizer for historical spelling variants using a transformer architecture.

GNU General Public License v3.0

6 stars 1 forks source link

Improve data handling for training and inference #80

Closed ybracke closed 8 months ago

ybracke commented 8 months ago

Functionality to perform sorting, filtering and chunked shuffling - should help with faster inference and training

https://pad.gwdg.de/voN43VoJSiSp5k27pgRBjA?view#Efficient-batching

TODOs

[x] Write tests for functions in data.process
[x] Add function filter_dataset_by_length to data.process?
[x] Integrate functions from data.process into models.generate
[x] Integrate functions from data.process into models.train_model