-
Migrate from Fixed Chunking of Texts to a more NLP based approach to chunking as referenced in https://vectara.com/blog/grounded-generation-done-right-chunking/
-
after try step from readme
```
curl -X POST http://127.0.0.1:8080/v1/create/rag -F "file=@paris.txt"
```
It took 590824.84 ms = nearly 1 minute for only chunking 306 lines (91KB) file on m3 max.
…
-
Since OpenSearch 2.13, [**fixed token length algorithm**](https://opensearch.org/docs/latest/ingest-pipelines/processors/text-chunking/#fixed-token-length-algorithm) is available in text chunking proc…
-
### What is the problem?
There is an image in the memory management section that has no background (transparent) and the text is black so can't be seen in dark mode: https://carpentries.github.io/ins…
-
**Is your feature request related to a problem? Please describe.**
I would like an optional flag for chunking strategy 'hi-res' so that tables are extracted but not images. Image text extracted as …
-
**Proposal**: Assess critical variables affecting semantic search quality on legal corpus before implementing semantic search on CL.
**Key Variables**:
- Embedding model
- Chunking strategy
- I…
-
This important, but difficult to explain - ask me if I am not clear.
Two 'granularities' are relevant when chunking the ENB reports
1. The first and coarser level is that of 'sections' corresponding…
tommv updated
8 years ago
-
**Describe the bug**
Sometimes when using chunking, the `text_as_html` for Table elements is missing some of the content compared to `text` property.
Reasoning:
- Text for a table can only come fro…
-
I Have the Clear pdf with proper images but this give
from unstructured.partition.pdf import partition_pdf
from PIL import UnidentifiedImageError
# Extract images, tables, and chunk text
…
-
Currently we do a character/word based chunking that is very simple. We should enhance our chunking strategies to possibly include:
* Recursive Character Chunking
* Token Based Chunking
* Documen…