-
Of all aspects challenging the readability of an argparse output for the 95% of us, or making people avoid reading too much, perhaps the density of the text is one of the worst sticking points. This i…
-
Embedding models have differing context lengths. `intfloat/multilingual-e5-base` for example has an input limit of 512 tokens. After that it just truncates the text.
Texts in MTEB's datasets are …
-
outline_writer.py
```
def chunking(self, papers, titles, chunk_size = 14000):
paper_chunks, title_chunks = [], []
...
paper_chunks.append(papers[start:])
title_ch…
-
**As a RAG Experiment Accelerator user** I would like to have a meaningful, hierarchy based config file structure
SO I would be able to **easily understand** the different features and settings which…
-
**Is your feature request related to a problem? Please describe.**
Currently, when processing PDF documents using the chunk_by_title function from the Unstructured library, a Table element always f…
-
### Question Validation
- [X] I have searched both the documentation and discord for an answer.
### Question
Hi everyone , I am benchmarking time it takes to chunk , embed documents and then insert…
-
### Problem statement
Currently, we are not able to train `SequenceTagger` models with tagged `Sentence` objects exceeding the token limit (typically 512). It does seem there is some support for long…
-
**Motivation:**
Right now, the chunking uses a greedy algorithm. The following would output the following chunks:
```rust
let text = "Sentence 1. Sentence 2. Sentence 3. Sentence 4.";
splitter.chunk…
-
I am using the Python bindings and noticed that text-splitter is running on one core only. I think it would be great to allow for an option to use all available CPU cores (at least for character-based…
do-me updated
2 weeks ago
-
Reasons for:
- 1 time investment. no more dealing with text stream overhead, only optimised operations.
- Respect the poly-indexability of our data. We can index with timestep, box-time or atom_id, …