dosubot[bot]
commented
3 months ago To address the issue with PDFReader when using an S3 file due to Path() casting, you can modify the load_data method to handle S3 paths correctly. Here is a solution using the s3fs library:
-
Ensure
s3fsis installed:pip install s3fs -
Modify the
load_datamethod inPDFReader: Update the method to handle S3 paths without casting them toPathobjects. Here is an example:import s3fs from PyPDF2 import PdfFileReader import io class PDFReader(BaseReader): """PDF parser.""" def __init__(self, return_full_document: Optional[bool] = False) -> None: self.return_full_document = return_full_document def load_data(self, file: str, extra_info: Optional[Dict] = None) -> List[Document]: """Parse file.""" try: import pypdf except ImportError: raise ImportError("pypdf is required to read PDF files: `pip install pypdf`") if file.startswith("s3://"): fs = s3fs.S3FileSystem() with fs.open(file, "rb") as fp: pdf = pypdf.PdfReader(fp) num_pages = len(pdf.pages) docs = self._extract_text_from_pdf(pdf, num_pages, fp.name, extra_info) else: with open(file, "rb") as fp: pdf = pypdf.PdfReader(fp) num_pages = len(pdf.pages) docs = self._extract_text_from_pdf(pdf, num_pages, fp.name, extra_info) return docs def _extract_text_from_pdf(self, pdf, num_pages, file_name, extra_info): docs = [] if self.return_full_document: text = "" metadata = {"file_name": file_name} for page in range(num_pages): page_text = pdf.pages[page].extract_text() text += page_text if extra_info is not None: metadata.update(extra_info) docs.append(Document(text=text, metadata=metadata)) else: for page in range(num_pages): page_text = pdf.pages[page].extract_text() page_label = pdf.page_labels[page] metadata = {"page_label": page_label, "file_name": file_name} if extra_info is not None: metadata.update(extra_info) docs.append(Document(text=page_text, metadata=metadata)) return docs
This approach ensures that S3 paths are handled correctly without being cast to Path objects, thus avoiding the WindowsPath issue [1].
To continue talking to Dosu, mention @dosu.
Chris-SL-Apps
logan-markewich
seeifsalem
wapiflapi
NguyenTuan9347
Bug Description
Hey, so to sum it up, I create a SimpleDirectoryReader with a PDFReader as an extractor and an s3 bucket as an input_dir, with also s3 as a fs. ThenI call load_data() which leads to the PDFReader's method load_data(file=input_dir). the input dir is in the s3 format but first thing in load_data are these 2 lines if not isinstance(file, Path): file = Path(file) This transforms it into a WindowsPath instead of a PosixPath and that doesn't work with S3. I think these two lines should be deleted or at least adapted to the filesystem at use PS: I opened this issue with the wrong github account minutes ago, don't mind that one, I closed it
Version
0.10.65
Steps to Reproduce
versions used are the most recent as of today 15.08.2024 for llama-index-readers-file llama-index and llama-index-core input_dir = f"{bucket}/{files_dir}" files_to_exclude = processed_files loader = SimpleDirectoryReader(input_dir=input_dir, exclude=files_to_exclude, file_extractor=extractors, fs=s3fs])
s3fs being an instance of s3fs.core.S3FileSystem
docs = loader.load_data()
Relevant Logs/Tracbacks