Closed mcognetta closed 1 week ago
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
rebasing on main should fix clippy issues
Thanks. I fixed one of the lint errors which was a range readability thing.
Fixed one more formatting issue. Now I think it should be all good!
I goofed, and now its getting worse 😬
Ok, it's all good now, unless you want me to squash the commits.
This is related to the discussion in #1541.
This PR allows for
0.0
to be used as the dropout value in BPE models with equivalent functionality tonone
. Previously, the docs and implementation were inconsistent:none
none
)dropout \in (0.0, 1.0]
BPE(dropout = 0.0)
)This simply allows for
0.0
to be an acceptable value during initialization and enables caching when tokenizing ifdropout == 0.0
.E.g., now the following works
whereas before it errored.
As future work, I think that dropout should be made non-optional, with the default being 0.0. This would remove the checks for
dropout.is_none()
, etc, but keep the functionality the same. However, I guess this would be a breaking change (since then all tokenizers serialized before this change would be invalid?).