Bump sentencepiece from 0.1.99 to 0.2.0

[dependabot[bot]]

Bumps sentencepiece from 0.1.99 to 0.2.0.

Release notes

Sourced from sentencepiece's releases.

v0.2.0

Major changes

N/A

New features

• [ALL] Added SentencePieceNormalizer class in C++/Python. It supports almost the equivalent feature of spm_normalize. Python Sample C++ Sample
• [ALL] Added SentencePieceProcessor::Normalize method in C++/Python Python Sample C++ Sample
• [ALL] Added functionality to override the normalization spec before the processing. Python Sample

Bug fixes & minor changes

• Introduce better support of using external abseil and protobuf google/sentencepiece#869
• Build universal binary in OSX release package google/sentencepiece#892
• Add the set_min_log_level function to python to change the loglevel from the python wrapper. google/sentencepiece#893
• Uses the logsumexp techniques in marginal probabilities of n-best tokenization to avoid underflow.
• Support Python 3.12 google/sentencepiece#932
• Improves the thread utilization in batch encoding/decoding.
• Fix nasty bug in BPE position encoding.
• Fix bugs in the handling of duplicated bigrams

v0.2.0pre1

Major changes

N/A

New features

• [ALL] Added SentencePieceNormalizer class in C++/Python. It supports almost the equivalent feature of spm_normalize. Python Sample C++ Sample
• [ALL] Added SentencePieceProcessor::Normalize method in C++/Python Python Sample C++ Sample
• [ALL] Added functionality to override the normalization spec before the processing. Python Sample

Bug fixes & minor changes

• Introduce better support of using external abseil and protobuf google/sentencepiece#869
• Build universal binary in OSX release package google/sentencepiece#892
• Add the set_min_log_level function to python to change the loglevel from the python wrapper. google/sentencepiece#893
• Uses the logsumexp techniques in marginal probabilities of n-best tokenization to avoid underflow.
• Support Python 3.12 google/sentencepiece#932
• Improves the thread utilization in batch encoding/decoding.
• Fix nasty bug in BPE position encoding.
• Fix bugs in the handling of duplicated bigrams

Commits

• 17d7580 suppress warnings in testharnress
• 4a3cd1c Merge pull request #975 from google/dependabot/pip/dot-github/workflows/requi...
• 670d2e7 Bump cryptography in /.github/workflows/requirements
• 2b8772a Merge pull request #974 from xunkai55/patch-1
• ffd8e9e Fix a typo in api.md
• 03243af Merge pull request #970 from google/dependabot/pip/dot-github/workflows/requi...
• d0fe405 Merge pull request #972 from google/dependabot/pip/dot-github/workflows/requi...
• a8a618f Bump cryptography in /.github/workflows/requirements
• bbbe548 Bump the build-time-deps group
• 53de765 allows to load precomputed seed sentencepieces for unigram from a file.
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)