search

openchatai / OpenChat

LLMs custom-chatbots console ⚡
https://open.cx
MIT License
5.17k stars 642 forks source link

PDF upload failed. How to upload size of PDF #47

Closed cidrugHug8 closed 1 year ago

cidrugHug8 commented 1 year ago

Hi, I would like to increase the upload size of my PDF files, can you tell me where to set the file size? Thank you.

Drugjudy commented 1 year ago

Hi, I would like to increase the upload size of my PDF files, can you tell me where to set the file size? Thank you.

The size limitation for parsing PDF files using pdf-parse would depend on the capabilities and limitations of the library itself. The code dynamically imports the pdf-parse library from the pdf-parse/lib/pdf-parse.js file. The specific limitations of pdf-parse may be documented in the library's documentation or source code.

llm-server/utils/customPDFLoader.ts

It's worth noting that processing very large PDF files can consume significant memory and processing resources. Therefore, even if there is no explicit limitation in the code, the performance and resource usage may be affected by the size and complexity of the PDF files being processed.

you can use this repo if this helps https://github.com/mayooear/gpt4-pdf-chatbot-langchain

cidrugHug8 commented 1 year ago

The size of the PDF file is 25,277KB. However, when I attempted to upload the PDF, I encountered an error and was unable to complete the upload. Here is the link to the error details: https://flareapp.io/share/q5YNGnom#context

Should I make changes to the nginx settings to resolve this issue?

cidrugHug8 commented 1 year ago

I managed to upload the PDF by deleting some pages and reducing their size. However, it seems that the text was not imported properly. The error from llm_server is as follows:

event - compiled successfully in 118 ms (55 modules)
[WARN] Importing from 'langchain/document_loaders' is deprecated. Import from eg. 'langchain/document_loaders/fs/text' or 'langchain/document_loaders/web/cheerio' instead. See https://js.langchain.com/docs/getting-started/install#updating-from-0052 for upgrade instructions.
Error
    at UnknownErrorExceptionClosure (/usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:443:37)
    at Object.<anonymous> (/usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:446:2)
    at __w_pdfjs_require__ (/usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:45:30)
    at Object.<anonymous> (/usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:7939:23)
    at __w_pdfjs_require__ (/usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:45:30)
    at /usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:88:18
    at /usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:91:10
    at webpackUniversalModuleDefinition (/usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:18:20)
    at Object.<anonymous> (/usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:25:3)
    at Module._compile (node:internal/modules/cjs/loader:1254:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1308:10)
    at Module.load (node:internal/modules/cjs/loader:1117:32)
    at Module._load (node:internal/modules/cjs/loader:958:12)
    at Module.require (node:internal/modules/cjs/loader:1141:19)
    at require (node:internal/modules/cjs/helpers:110:18)
    at PDF (/usr/src/app/node_modules/pdf-parse/lib/pdf-parse.js:63:29)
    at CustomPDFLoader.parse (webpack-internal:///(api)/./utils/customPDFLoader.ts:42:30)
    at async DirectoryLoader.load (file:///usr/src/app/node_modules/langchain/dist/document_loaders/fs/directory.js:66:40)
    at async pdfHandler (webpack-internal:///(api)/./data-sources/pdfHandler.ts:29:25)
    at async handler (webpack-internal:///(api)/./pages/api/ingest.ts:38:20)
    at async Object.apiResolver (/usr/src/app/node_modules/next/dist/server/api-utils/node.js:372:9)
    at async DevServer.runApi (/usr/src/app/node_modules/next/dist/server/next-server.js:513:9)
    at async Object.fn (/usr/src/app/node_modules/next/dist/server/next-server.js:815:35)
    at async Router.execute (/usr/src/app/node_modules/next/dist/server/router.js:243:32)
    at async DevServer.runImpl (/usr/src/app/node_modules/next/dist/server/base-server.js:432:29)
    at async DevServer.run (/usr/src/app/node_modules/next/dist/server/dev/next-dev-server.js:814:20)
    at async DevServer.handleRequestImpl (/usr/src/app/node_modules/next/dist/server/base-server.js:375:20)
    at async /usr/src/app/node_modules/next/dist/server/base-server.js:157:99 {
  message: 'PDFDocument: stream must have data',
  details: 'Error: PDFDocument: stream must have data'
}
buildnewapp commented 1 year ago

I managed to upload the PDF by deleting some pages and reducing their size. However, it seems that the text was not imported properly. The error from llm_server is as follows:

event - compiled successfully in 118 ms (55 modules)
[WARN] Importing from 'langchain/document_loaders' is deprecated. Import from eg. 'langchain/document_loaders/fs/text' or 'langchain/document_loaders/web/cheerio' instead. See https://js.langchain.com/docs/getting-started/install#updating-from-0052 for upgrade instructions.
Error
    at UnknownErrorExceptionClosure (/usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:443:37)
    at Object.<anonymous> (/usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:446:2)
    at __w_pdfjs_require__ (/usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:45:30)
    at Object.<anonymous> (/usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:7939:23)
    at __w_pdfjs_require__ (/usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:45:30)
    at /usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:88:18
    at /usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:91:10
    at webpackUniversalModuleDefinition (/usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:18:20)
    at Object.<anonymous> (/usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:25:3)
    at Module._compile (node:internal/modules/cjs/loader:1254:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1308:10)
    at Module.load (node:internal/modules/cjs/loader:1117:32)
    at Module._load (node:internal/modules/cjs/loader:958:12)
    at Module.require (node:internal/modules/cjs/loader:1141:19)
    at require (node:internal/modules/cjs/helpers:110:18)
    at PDF (/usr/src/app/node_modules/pdf-parse/lib/pdf-parse.js:63:29)
    at CustomPDFLoader.parse (webpack-internal:///(api)/./utils/customPDFLoader.ts:42:30)
    at async DirectoryLoader.load (file:///usr/src/app/node_modules/langchain/dist/document_loaders/fs/directory.js:66:40)
    at async pdfHandler (webpack-internal:///(api)/./data-sources/pdfHandler.ts:29:25)
    at async handler (webpack-internal:///(api)/./pages/api/ingest.ts:38:20)
    at async Object.apiResolver (/usr/src/app/node_modules/next/dist/server/api-utils/node.js:372:9)
    at async DevServer.runApi (/usr/src/app/node_modules/next/dist/server/next-server.js:513:9)
    at async Object.fn (/usr/src/app/node_modules/next/dist/server/next-server.js:815:35)
    at async Router.execute (/usr/src/app/node_modules/next/dist/server/router.js:243:32)
    at async DevServer.runImpl (/usr/src/app/node_modules/next/dist/server/base-server.js:432:29)
    at async DevServer.run (/usr/src/app/node_modules/next/dist/server/dev/next-dev-server.js:814:20)
    at async DevServer.handleRequestImpl (/usr/src/app/node_modules/next/dist/server/base-server.js:375:20)
    at async /usr/src/app/node_modules/next/dist/server/base-server.js:157:99 {
  message: 'PDFDocument: stream must have data',
  details: 'Error: PDFDocument: stream must have data'
}

The same as me

gharbat commented 1 year ago

This happens when you upload non-scannable PDF files (contain images not text).

psociety commented 1 year ago

@gharbat this also happens on pdfs that have both, text and images. There are some ocr solutions that overlap the text on a new layer so you still see the image but the pdf can be searched and the text selected. Example: https://github.com/ocrmypdf/OCRmyPDF