dosubot[bot]
commented
1 month ago Hey @truski22! 👋 I'm here to assist you with any bugs, questions, or contributions while you wait for a human maintainer. Let's squash those bugs together!
To ensure consistent hyperlink handling both inside and outside the IngestionPipeline when using the splitter.split_text_metadata_aware method, you need to customize the tokenizer or the splitting functions to recognize and preserve hyperlinks.
Here's an example of how you might modify the SentenceSplitter class to handle hyperlinks consistently:
- Custom Tokenizer: Modify the tokenizer to treat hyperlinks as single tokens.
- Custom Split Functions: Ensure that the split functions do not break hyperlinks.
import re
from typing import List, Optional, Callable
from llama_index.core.node_parser.text.sentence import SentenceSplitter
from llama_index.core.callbacks.base import CallbackManager
from llama_index.core.schema import Document
def custom_tokenizer(text: str) -> List[str]:
"""Custom tokenizer that treats hyperlinks as single tokens."""
# Regex to match hyperlinks
hyperlink_regex = r'https?://\S+|www\.\S+'
tokens = re.findall(hyperlink_regex, text)
# Split the text by spaces and add the hyperlinks
return re.split(r'(\s+)', text) + tokens
class CustomSentenceSplitter(SentenceSplitter):
def __init__(
self,
chunk_size: int = SentenceSplitter.chunk_size,
chunk_overlap: int = SentenceSplitter.chunk_overlap,
tokenizer: Optional[Callable] = custom_tokenizer,
callback_manager: Optional[CallbackManager] = None,
separator: str = " ",
paragraph_separator: str = SentenceSplitter.paragraph_separator,
secondary_chunking_regex: str = SentenceSplitter.secondary_chunking_regex,
include_metadata: bool = True,
include_prev_next_rel: bool = True,
id_func: Optional[Callable[[int, Document], str]] = None,
):
super().__init__(
chunk_size=chunk_size,
chunk_overlap=chunk_overlap,
tokenizer=tokenizer,
callback_manager=callback_manager,
separator=separator,
paragraph_separator=paragraph_separator,
secondary_chunking_regex=secondary_chunking_regex,
include_metadata=include_metadata,
include_prev_next_rel=include_prev_next_rel,
id_func=id_func,
)
def _split(self, text: str, chunk_size: int) -> List[str]:
"""Override _split to ensure hyperlinks are not broken."""
if len(self._tokenizer(text)) <= chunk_size:
return [text]
for split_fn in self._split_fns:
splits = split_fn(text)
if len(splits) > 1:
break
new_splits = []
for split in splits:
split_len = len(self._tokenizer(split))
if split_len <= chunk_size:
new_splits.append(split)
else:
# recursively split
new_splits.extend(self._split(split, chunk_size=chunk_size))
return new_splits
# Usage
splitter = CustomSentenceSplitter()
chunks = splitter.split_text_metadata_aware(text, metadata_str)In this example, the custom_tokenizer function uses a regular expression to identify hyperlinks and treat them as single tokens. The CustomSentenceSplitter class then uses this tokenizer to ensure that hyperlinks are not broken during the splitting process. This approach ensures consistent handling of hyperlinks both inside and outside the IngestionPipeline [1][2][3][4].
To continue talking to Dosu, mention @dosu.
truski22
Question Validation
Question
I am implementing a sentence splitter for texts. If the text contains hyperlinks, it behaves differently if I run the splitter within an IngestionPipeline compared to running it outside the IngestionPipeline as follows: file_splitted = splitter.split_text_metadata_aware(text, node.get_metadata_str()) Hyperlinks display differently but the rest of the text doesn't, I'm confused about this behavior.