There is a minor bug in `DocumentLoader._load_document()`

User Experience:

While reading directories that include document types not supported by the application. GPT-Researcher crashes.

The Problem:

The function must return an iterable,
- which is what the except Exception handler is trying to manage by including a return [].
However, if loader = loader_dict.get(file_extension, None) returns None,
- which it will if the file_extension is not in the loader_dict dictionary.
- the function is successfully completed, without ever hitting a return statement thus by default it returns None.

The Solution:

The simplest solution would be to add an else on the if loader

if loader:
data = loader.load()
return data
else:
 return [] # Fixes Bug

My personal preference would be if your method is expected to return an iterable.

Make that return the last line of the method, and don't allow other return paths.

Rationale: This can help in maintainability as someone who does not know the code can immediately see that it returns an iterable.


async def _load_document(self, file_path: str, file_extension: str) -> list:
ret_data = []
try:
    loader_dict = {
        "pdf": PyMuPDFLoader(file_path),
        "txt": TextLoader(file_path),
        "doc": UnstructuredWordDocumentLoader(file_path),
        "docx": UnstructuredWordDocumentLoader(file_path),
        "pptx": UnstructuredPowerPointLoader(file_path),
        "csv": UnstructuredCSVLoader(file_path, mode="elements"),
        "xls": UnstructuredExcelLoader(file_path, mode="elements"),
        "xlsx": UnstructuredExcelLoader(file_path, mode="elements"),
        "md": UnstructuredMarkdownLoader(file_path)
    }

    loader = loader_dict.get(file_extension, None)
    if loader:
        ret_data = loader.load()

except Exception as e:
    print(f"Failed to load document : {file_path}")
    print(e)

return ret_data



Note: I'm happy to do a PR, but the fix is simple, and I was hoping other folks actively coding might just slip it in.

Thanks to all who are building such a great product here!!

assafelovic / gpt-researcher

There is a minor bug in `DocumentLoader._load_document()` #555