Closed stephentoub closed 1 month ago
All modified and coverable lines are covered by tests :white_check_mark:
Project coverage is 68.78%. Comparing base (
be1e428
) to head (14fff61
). Report is 5 commits behind head on main.
/ba-g unrelated torchsharp crash
This updates two of the regexes to match the changes made in https://github.com/openai/tiktoken/commit/9f7f69d62d6052dcc2fd54357df6ae9ae2590518.
On .NET Core, these changes are mostly nops, as the main thing they're doing is changing some loops to be atomic, and the auto-atomicity logic in the regex optimizer was already noticing that could be done and doing it automatically. On .NET Framework, it's a bigger deal, as those loops will now be atomic where they weren't previously.
If nothing else, it keeps the regexes in sync with the reference implementation.