Open jawar-cn opened 1 month ago
The issue is likely due to the text_as_html property in the metadata, which is a large object and is being deep copied unnecessarily.
To fix this, you can try the following , you can try setting infer_table_structure=False when creating the UnstructuredWordDocumentLoader, like this:
loader = UnstructuredWordDocumentLoader(self.file_path, mode="paged", strategy="fast", infer_table_structure=False)
This will prevent the text_as_html property from being generated and reduce the memory usage.
if this issue presists , comment on this issue again and I will further look into this issue
Thanks , And yes, that's what I tried, and it fixed, and I think when deepcopy metadata, some case may need to selectively ignore some attributes. in case of sometime need to set infer_table_structure=True and split text at the same time.
Glad to hear that 😊
Checked other resources
Example Code
Error Message and Stack Trace (if applicable)
No response
Description
Use
from langchain_community.document_loaders import UnstructuredWordDocumentLoader
to parse word file with big table, then gotDocument
objects. if paraminfer_table_structure=True
which is default param, each document metadata property contains atext_as_html
property which is a big object. Then when usingTextSplitter
to split documents into chunks, every chunks document will deepcopy metadata once. If there are many chunks, the memory will increase sharply, and finally oom, and the program will terminate.System Info
langchain==0.2.3 langchain-community==0.2.4 langchain-core==0.2.5 langchain-text-splitters==0.2.1