Split text into semantic chunks, up to a desired chunk size. Supports calculating length by characters and tokens, and is callable from Rust and Python.
Finally, the so long awaited math mode is here! Enable the option to use it.
This release also includes other improvements and bugfixes, please see the changelog below for more details. Thanks to all contributors that has made possible this release!
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself)
- `@dependabot ignore minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself)
- `@dependabot ignore ` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore ` will remove all of the ignore conditions of the specified dependency
- `@dependabot unignore ` will remove the ignore condition of the specified dependency and ignore conditions
Bumps the minor group with 6 updates:
0.10.3
0.11.0
0.5.8
0.5.9
0.20.8
0.20.9
0.20.8
0.20.9
0.20.8
0.20.9
0.10.0
0.11.1
Updates
pulldown-cmark
from 0.10.3 to 0.11.0Release notes
Sourced from pulldown-cmark's releases.
Commits
d7632ac
chore: update version to 0.11.0 and Cargo.lock57b6b3f
Merge pull request #887 from pulldown-cmark/branch_0.117137d4e
Merge branch 'master' into branch_0.11a9a6ed7
Merge pull request #885 from tomcur/remove-unnecessary-set8fde095
fix: remove unnecessary end_newline set056c9e1
Merge pull request #880 from ollpu/escape-simd7316da2
(Re)introduce simd feature to escape crate7a1e53c
Merge pull request #870 from stepantubanov/write-to-fmtcefabf3
do not reexport writer wrappers00c9c48
Update pulldown-cmark-escape/src/lib.rsUpdates
tiktoken-rs
from 0.5.8 to 0.5.9Release notes
Sourced from tiktoken-rs's releases.
Commits
35ae83e
Bump version patch to "0.5.9"6f7528a
nit: Appease lint8ce2f09
Add support: o200k_base tokenizer.eb9c1de
Add text embedding models14821ea
Update README.mdbf3adc9
fix: Update context size for multiple models in model.rsa88091f
Addgpt-3.5-turbo-0125
(#60)d9dcfd8
[chore] Fix lintf84907c
Addgpt-4-0125
andgpt-3.5-turbo-1106
context sizes (#59)Updates
darling
from 0.20.8 to 0.20.9Changelog
Sourced from darling's changelog.
Commits
Updates
darling_core
from 0.20.8 to 0.20.9Changelog
Sourced from darling_core's changelog.
Commits
Updates
darling_macro
from 0.20.8 to 0.20.9Changelog
Sourced from darling_macro's changelog.
Commits
Updates
strsim
from 0.10.0 to 0.11.1Release notes
Sourced from strsim's releases.
Changelog
Sourced from strsim's changelog.
Commits
f72cd1c
publish 0.11.1680fa50
Use rust-version field of Cargo.toml (#72)d84fd91
Drop MSRV down to 1.56 (#71)7beabcb
fix ci workflow warnings (#70)f71334a
Create SECURITY.md09fd5d9
fix version numbere7a3b55
Update the installation instructionsd9f10d5
release version 0.10.1a5cc615
add maintainer20e3bdd
only boost similarity in Jaro-Winkler once the Jaro similarity exceeds 0.7Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting
@dependabot rebase
.Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show