After a successfull launch, does not reads the PDF loaded

[mbaggnx]

Hello there!

I already managed to launch and make it work, but now I am facing a problem where the langchain is not reading the PDF file

I am using thise LLM: all_datasets_v4_MiniLM-L6 vicuna-7b-1.1 vicuna-13b wizardLM-7B-HF WizardLM-7B-uncensored-GPTQ

Launching the comand of oobabooga by CPU (no auto devieces since I dont have GPU)

Any clue or somehow to research about it?

Some screenshots:

[mbaggnx]

Updated, it seems keeps working, filling up the HDD, but does not gives any response...

[mbaggnx]

I am going to update the topic by the work over the reddit forum: https://www.reddit.com/r/LangChain/comments/13cg8lp/comment/jsgd9mz/?context=3

Finaly, added to the code:

` from langchain.llms import TextGen

llm = TextGen(
        model_url='http://127.0.0.1:7860',
        temperature=0.1,
        max_new_tokens=2048,
        top_p=0.1,
        top_k=40,
        stopping_strings=['### System:', '### User:', '\n\n']
    )
`

But it does not work

Keep analizyng why

[mbaggnx]

Afert an update of the langchain, still keep not working, working on it still.

Updated to Langchain langchain==0.0.235 and then:

(C:\LangChain_PDFChat_Oobabooga-main\installer_files\env) C:\LangChain_PDFChat_Oobabooga-main>python -m pip install --upgrade langchain Requirement already satisfied: langchain in c:\langchain_pdfchat_oobabooga-main\installer_files\env\lib\site-packages (0.0.235) Collecting langchain Using cached langchain-0.0.240-py3-none-any.whl (1.4 MB) Requirement already satisfied: PyYAML>=5.4.1 in c:\langchain_pdfchat_oobabooga-main\installer_files\env\lib\site-packages (from langchain) (6.0.1) Requirement already satisfied: SQLAlchemy<3,>=1.4 in c:\langchain_pdfchat_oobabooga-main\installer_files\env\lib\site-packages (from langchain) (2.0.19) Requirement already satisfied: aiohttp<4.0.0,>=3.8.3 in c:\langchain_pdfchat_oobabooga-main\installer_files\env\lib\site-packages (from langchain) (3.8.5) Requirement already satisfied: async-timeout<5.0.0,>=4.0.0 in c:\langchain_pdfchat_oobabooga-main\installer_files\env\lib\site-packages (from langchain) (4.0.2) Requirement already satisfied: dataclasses-json<0.6.0,>=0.5.7 in c:\langchain_pdfchat_oobabooga-main\installer_files\env\lib\site-packages (from langchain) (0.5.13) Collecting langsmith<0.1.0,>=0.0.11 (from langchain) Using cached langsmith-0.0.14-py3-none-any.whl (29 kB) Requirement already satisfied: numexpr<3.0.0,>=2.8.4 in c:\langchain_pdfchat_oobabooga-main\installer_files\env\lib\site-packages (from langchain) (2.8.4) Requirement already satisfied: numpy<2,>=1 in c:\langchain_pdfchat_oobabooga-main\installer_files\env\lib\site-packages (from langchain) (1.24.4) Requirement already satisfied: openapi-schema-pydantic<2.0,>=1.2 in c:\langchain_pdfchat_oobabooga-main\installer_files\env\lib\site-packages (from langchain) (1.2.4) Requirement already satisfied: pydantic<2,>=1 in c:\langchain_pdfchat_oobabooga-main\installer_files\env\lib\site-packages (from langchain) (1.10.11) Requirement already satisfied: requests<3,>=2 in c:\langchain_pdfchat_oobabooga-main\installer_files\env\lib\site-packages (from langchain) (2.31.0) Requirement already satisfied: tenacity<9.0.0,>=8.1.0 in c:\langchain_pdfchat_oobabooga-main\installer_files\env\lib\site-packages (from langchain) (8.2.2) Requirement already satisfied: attrs>=17.3.0 in c:\langchain_pdfchat_oobabooga-main\installer_files\env\lib\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (23.1.0) Requirement already satisfied: charset-normalizer<4.0,>=2.0 in c:\langchain_pdfchat_oobabooga-main\installer_files\env\lib\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (3.2.0) Requirement already satisfied: multidict<7.0,>=4.5 in c:\langchain_pdfchat_oobabooga-main\installer_files\env\lib\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (6.0.4) Requirement already satisfied: yarl<2.0,>=1.0 in c:\langchain_pdfchat_oobabooga-main\installer_files\env\lib\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.9.2) Requirement already satisfied: frozenlist>=1.1.1 in c:\langchain_pdfchat_oobabooga-main\installer_files\env\lib\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.4.0) Requirement already satisfied: aiosignal>=1.1.2 in c:\langchain_pdfchat_oobabooga-main\installer_files\env\lib\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.3.1) Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in c:\langchain_pdfchat_oobabooga-main\installer_files\env\lib\site-packages (from dataclasses-json<0.6.0,>=0.5.7->langchain) (3.20.1) Requirement already satisfied: typing-inspect<1,>=0.4.0 in c:\langchain_pdfchat_oobabooga-main\installer_files\env\lib\site-packages (from dataclasses-json<0.6.0,>=0.5.7->langchain) (0.9.0) Requirement already satisfied: typing-extensions>=4.2.0 in c:\langchain_pdfchat_oobabooga-main\installer_files\env\lib\site-packages (from pydantic<2,>=1->langchain) (4.5.0) Requirement already satisfied: idna<4,>=2.5 in c:\langchain_pdfchat_oobabooga-main\installer_files\env\lib\site-packages (from requests<3,>=2->langchain) (3.4) Requirement already satisfied: urllib3<3,>=1.21.1 in c:\langchain_pdfchat_oobabooga-main\installer_files\env\lib\site-packages (from requests<3,>=2->langchain) (1.26.16) Requirement already satisfied: certifi>=2017.4.17 in c:\langchain_pdfchat_oobabooga-main\installer_files\env\lib\site-packages (from requests<3,>=2->langchain) (2023.7.22) Requirement already satisfied: greenlet!=0.4.17 in c:\langchain_pdfchat_oobabooga-main\installer_files\env\lib\site-packages (from SQLAlchemy<3,>=1.4->langchain) (2.0.2) Requirement already satisfied: packaging>=17.0 in c:\langchain_pdfchat_oobabooga-main\installer_files\env\lib\site-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses-json<0.6.0,>=0.5.7->langchain) (23.1) Requirement already satisfied: mypy-extensions>=0.3.0 in c:\langchain_pdfchat_oobabooga-main\installer_files\env\lib\site-packages (from typing-inspect<1,>=0.4.0->dataclasses-json<0.6.0,>=0.5.7->langchain) (1.0.0) Installing collected packages: langsmith, langchain Attempting uninstall: langsmith Found existing installation: langsmith 0.0.7 Uninstalling langsmith-0.0.7: Successfully uninstalled langsmith-0.0.7 Attempting uninstall: langchain Found existing installation: langchain 0.0.235 Uninstalling langchain-0.0.235: Successfully uninstalled langchain-0.0.235 Successfully installed langchain-0.0.240 langsmith-0.0.14

[makiJanus]

I have the same issue running the model on GPU, it processes and loads the knowledge base, but it seems like ignoring it, the only (superficial) change that I can see from the original ask-pdf code is that is no using callbacks when defining the llm model, so I'll try to check if that's important or not.

Updates:

1. Callback checked, it only stream the output, so its fine not using it. Now trying to see if the pdf information is ok.
2. PDF loading and similarity research are ok, now looking into the chain process.
3. chain functions are fine, checking other parts of the code.

My workaround: After checking the chain methods, I noticed that I was getting the following warning "Token indices sequence length is longer than the specified maximum sequence length for this model (1124 > 1024). Running this sequence through the model will result in indexing errors". So I changed the chunk size to 700 and the chunk overlap to 100 in the tex_splitter variable, after that I works flawless, another way to solve it may be increasing the maximum sequence length, I'll check that too.

[mbaggnx]

I have the same issue running the model on GPU, it processes and loads the knowledge base, but it seems like ignoring it, the only (superficial) change that I can see from the original ask-pdf code is that is no using callbacks when defining the llm model, so I'll try to check if that's important or not.

Updates:

1. Callback checked, it only stream the output, so its fine not using it. Now trying to see if the pdf information is ok.
2. PDF loading and similarity research are ok, now looking into the chain process.
3. chain functions are fine, checking other parts of the code.

My workaround: After checking the chain methods, I noticed that I was getting the following warning "Token indices sequence length is longer than the specified maximum sequence length for this model (1124 > 1024). Running this sequence through the model will result in indexing errors". So I changed the chunk size to 700 and the chunk overlap to 100 in the tex_splitter variable, after that I works flawless, another way to solve it may be increasing the maximum sequence length, I'll check that too.

Hi there, thanks for the research, I am going to test your change on the chunk Also, which LLMs are you using for this? To understand, becuase I can load diferent ones than the one used

Thanks, kind regards!

[makiJanus]

I have the same issue running the model on GPU, it processes and loads the knowledge base, but it seems like ignoring it, the only (superficial) change that I can see from the original ask-pdf code is that is no using callbacks when defining the llm model, so I'll try to check if that's important or not. Updates:

1. Callback checked, it only stream the output, so its fine not using it. Now trying to see if the pdf information is ok.
2. PDF loading and similarity research are ok, now looking into the chain process.
3. chain functions are fine, checking other parts of the code.

My workaround: After checking the chain methods, I noticed that I was getting the following warning "Token indices sequence length is longer than the specified maximum sequence length for this model (1124 > 1024). Running this sequence through the model will result in indexing errors". So I changed the chunk size to 700 and the chunk overlap to 100 in the tex_splitter variable, after that I works flawless, another way to solve it may be increasing the maximum sequence length, I'll check that too.

Hi there, thanks for the research, I am going to test your change on the chunk Also, which LLMs are you using for this? To understand, becuase I can load diferent ones than the one used

Thanks, kind regards!

Sure! I tested with guanaco-7b-HF (https://huggingface.co/TheBloke/guanaco-7B-HF/tree/main) and llama-13b-4bit-128g (https://huggingface.co/Neko-Institute-of-Science/LLaMA-13B-4bit-128g). While checking the length issue I make sure that in the Oobabooga the max prompt size where up to 2048, so I think its possible that the problem may lay in:

• Chain configuration.
• Oobabooga args (specified the initialization with a max token size, I don't know if there is an arg for it xd).
• The model itself.

[mironkraft]

Hello there

I am experiencing the same issue and no clue after this days

Any chance to know if you have checked and progressed? @sebaxzero @makiJanus @mbaggnx ?

Thanks, kind regards

[makiJanus]

@mironkraft For me is already working reducing the maximum sequence length, what kind of troubles are you having? Are you trying a local implementation on gpu or cpu?

[mironkraft]

@mironkraft For me is already working reducing the maximum sequence length, what kind of troubles are you having? Are you trying a local implementation on gpu or cpu?

Hi there! I am on local with CPU/RAM Not able right now to use a GPU You made it over GPU?

Can you tell me exactly the code you changed? Maybe I changed it wrong...

Thanks for your fast reply

[makiJanus]

@mironkraft Really? This repo is intended to run on GPU, as the Readme says, Its a fork from another git that uses local llama on cpu, but this one uses Oobabooga yo interface with other models with Its api. Do you have Oobabooga installed? With that you can run on gpu, I recommend you yo re-read the first part of the Readme where the Oobabooga repo link is. Also near to it there os the repo from waffle that runs on cpu llama by default.

[mironkraft]

@mironkraft Really? This repo is intended to run on GPU, as the Readme says, Its a fork from another git that uses local llama on cpu, but this one uses Oobabooga yo interface with other models with Its api. Do you have Oobabooga installed? With that you can run on gpu, I recommend you yo re-read the first part of the Readme where the Oobabooga repo link is. Also near to it there os the repo from waffle that runs on cpu llama by default.

Yes, I already run Oobabooga with CPU/RAM. I do interface API the langchain with Oobabooga and run under the CPU

No GPU in any case

It works and hace the same error...

Your telling me that I can't use Oobabooga under CPU with LanchainPDF?

Never heard about Waffle, ill check

Thanks again for answering

[makiJanus]

@mironkraft Really? This repo is intended to run on GPU, as the Readme says, Its a fork from another git that uses local llama on cpu, but this one uses Oobabooga yo interface with other models with Its api. Do you have Oobabooga installed? With that you can run on gpu, I recommend you yo re-read the first part of the Readme where the Oobabooga repo link is. Also near to it there os the repo from waffle that runs on cpu llama by default.

Yes, I already run Oobabooga with CPU/RAM. I do interface API the langchain with Oobabooga and run under the CPU

No GPU in any case

It works and hace the same error...

Your telling me that I can't use Oobabooga under CPU with LanchainPDF?

Never heard about Waffle, ill check

Thanks again for answering

Hmm I see, lets try to find where the problem is, yes, it is possible to run Ooba on cpu as well. Which exact error do you have? What message is popping up when you try to run this repo? The error appears when you start Ooba or this repo?

[mironkraft]

@mironkraft Really? This repo is intended to run on GPU, as the Readme says, Its a fork from another git that uses local llama on cpu, but this one uses Oobabooga yo interface with other models with Its api. Do you have Oobabooga installed? With that you can run on gpu, I recommend you yo re-read the first part of the Readme where the Oobabooga repo link is. Also near to it there os the repo from waffle that runs on cpu llama by default.

Yes, I already run Oobabooga with CPU/RAM. I do interface API the langchain with Oobabooga and run under the CPU

No GPU in any case

It works and hace the same error...

Your telling me that I can't use Oobabooga under CPU with LanchainPDF?

Never heard about Waffle, ill check

Thanks again for answering

Hmm I see, lets try to find where the problem is, yes, it is possible to run Ooba on cpu as well. Which exact error do you have? What message is popping up when you try to run this repo? The error appears when you start Ooba or this repo?

The error appears when I introduce a question to the pdf, there is no other error I already fixed as you did the source files and nothing

Yes it is possible to run on CPU already dis achieve it.

This is my configuration of Oobabooga once model loaded

[mbaggnx]

Just to let you know, I am on CPU/RAM only mode too

Loading this model I got this error: https://huggingface.co/clibrain/Llama-2-ft-instruct-es/tree/main

Traceback (most recent call last): File “C:\LangChain_PDFChat_Oobabooga-main\text-generation-webui\[server.py](http://server.py/)”, line 68, in load_model_wrapper shared.model, shared.tokenizer = load_model(shared.model_name, loader) File “C:\LangChain_PDFChat_Oobabooga-main\text-generation-webui\modules\[models.py](http://models.py/)”, line 87, in load_model tokenizer = load_tokenizer(model_name, model) File “C:\LangChain_PDFChat_Oobabooga-main\text-generation-webui\modules\[models.py](http://models.py/)”, line 104, in load_tokenizer tokenizer = AutoTokenizer.from_pretrained( File “C:\LangChain_PDFChat_Oobabooga-main\installer_files\env\lib\site-packages\transformers\models\auto\tokenization_auto.py”, line 702, in from_pretrained return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs) File “C:\LangChain_PDFChat_Oobabooga-main\installer_files\env\lib\site-packages\transformers\tokenization_utils_base.py”, line 1841, in from_pretrained return cls.from_pretrained( File “C:\LangChain_PDFChat_Oobabooga-main\installer_files\env\lib\site-packages\transformers\tokenization_utils_base.py”, line 2004, in from_pretrained tokenizer = cls(*init_inputs, **init_kwargs) File “C:\LangChain_PDFChat_Oobabooga-main\installer_files\env\lib\site-packages\transformers\models\llama\tokenization_llama.py”, line 144, in init self.sp_model.Load(vocab_file) File "C:\LangChain_PDFChat_Oobabooga-main\installer_files\env\lib\site-packages\sentencepiece_init.py", line 905, in Load return self.LoadFromFile(model_file) File "C:\LangChain_PDFChat_Oobabooga-main\installer_files\env\lib\site-packages\sentencepiece_init.py", line 310, in LoadFromFile return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg) TypeError: not a string

[makiJanus]

@mironkraft For what I can see in your screenshot you have one warn and one error, for what I understand you can run it on CPU without problems, so lets try to figure it out what the cmd console says:

Warn: The instaled version of bitsandBytes is installet without GPU support, for what I could find here this is a library to wrapper cuda custom functions, so if you have installed without cuda, or you don't have cuda installed, it makes sense that it warns you about it. Here is a git issue with someone sharing the same problem as you, some users shared their solutions, remember that those may or not work in your case, but is always worth trying, these things happens quite a lot when we work with code xd.

Error: <Response [404]>, Oobabooga API works with an http API, so you the codes that it proves are related to that protocol, for example the code status 200 means that the request arrived successfully, but the 404 is, if I remember well, for a non existent endpoint, that means probably Ask-your-pdf is not connecting correctly to Oobabooga, it may be ralated to the warm some ip thing (less probably)

If you can Use Oobabooga alone, then the problem lies in the Ask-yout-pdf, maybe due start touching the code randomly or something like that. The way that i run this (works on the first try, then I fix the knowledge retrieval thing) was installing Oobabooga first, make sure that is working, and the download this repo, that is just and interface and pdf handler with langchain.

I run Oobabooga from the files in its repe, and the I do a "run streamlit app.py" (app is the name of my file) to run the ask-pdf interface, prabably some error may be derived from that, @mbaggnx may be useful to re-download this repo, following this approach.

I have a repo derived from this with additional options, maybe you can have some insights of how I run the thing https://github.com/makiJanus/pdf-chat

[mbaggnx]

Hi, thanks for reply. This is what I worked out

1- Installed, but same thing, tested on 2 laptops, doesnt matter since there is no GPU

2- Error 404, tested 2 types of API on Oobabooga, none works 2.1 - Tested changing the api link on Main.py (tested some variants), failure, screenshots:

Errors on Oobabooga

2.1.1 Second Test, trying and revising the API conection, none for now, I only get this message when I visit: http://127.0.0.1:5005/api/v1/stream

3- Installeg again Oobabooga with https://github.com/makiJanus/pdf-chat First attemp, failed (using oobabooga env)

(the window that opens it is not similar to the downloaded repo when execute strealit)

3.1- Updating Streamlit, launching again File "C:\Users\mXXXXXXXXXX\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\streamlit\runtime\scriptrunner\script_runner.py", line 552, in _run_script exec(code, module.__dict__) File "C:\pdf-chat-main\app.py", line 16, in <module> import streamlit_toggle as tog # Custom Streamlit widget for toggle switch ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

What I am doing wrong?

3.2 Fresh installed Oobabooga with fresh install of your repo,

when you say: ` 2.- Open start_windows.bat 3.- Run this git in your python environment.

cd pdf-chat streamlit run app.py ` "Run this git on your python environment" you mean the one generated by the CMD that launches Ooba right?

and that it is?

3.2.1 Tested is not working streamlit

even with 2 diferent calls, the one you posted, the one I created: python -m streamlit run app.py

3.2.2 Installed streamlit on the fresh env after the fail of launching both commands for run app.py (most of the requirements satisfied, and some other minor installs)

3.2.3 Since not working, I forced the launch from VSCode, and back to the 3.1 error again....

What I am doing wrong?

Thanks, kind regards!