Fixing tokenizers to correctly handle linux line endings (\n) - Githubissues

NMZivkovic / BertTokenizers

Open source project for BERT Tokenizers in C#.

MIT License

83 stars 22 forks source link

Fixing tokenizers to correctly handle linux line endings (\n) #25

Open palenshus opened 1 year ago

palenshus commented 1 year ago

Fixes https://github.com/NMZivkovic/BertTokenizers/issues/24