dotnet / machinelearning

ML.NET is an open source and cross-platform machine learning framework for .NET.
https://dot.net/ml
MIT License
8.92k stars 1.86k forks source link

Bpe.UnknownToken is mutable #7006

Closed stephentoub closed 4 months ago

stephentoub commented 4 months ago

Should it be? Changing it post construction can change the results of tokenizing. If this is supposed to be something created as a singleton and shared, UnknownToken should likely not be allowed to be changed post construction (either only as a ctor argument or at least making the property's setter instead be init).

stephentoub commented 4 months ago

cc: @tarekgh