Refactor data processing in train_model

ybracke / transnormer

A lexical normalizer for historical spelling variants using a transformer architecture.

GNU General Public License v3.0

6 stars 1 forks source link

Closed ybracke closed 8 months ago

ybracke commented 8 months ago

New and updated functions in train_model for data handling
- max_length are missing for now, option to filter out data of specific length (before training starts) should be added
Tests for the new and modified functions in test_train_model
Additional testdata file