search

PromtEngineer / localGPT

Chat with your documents on your local device using GPT models. No data leaves your device and 100% private.
Apache License 2.0
19.91k stars 2.22k forks source link

ingest.py is not running / mbp16 m1 #81

Open pascalandy opened 1 year ago

pascalandy commented 1 year ago

Error:

~/Documents/github_pascalandy/localGPT|main⚡ ⇒  python ingest.py
Traceback (most recent call last):
  File "/Users/andy16/Documents/github_pascalandy/localGPT/ingest.py", line 4, in <module>
    from utils import xlxs_to_csv
  File "/Users/andy16/Documents/github_pascalandy/localGPT/utils.py", line 1, in <module>
    import openpyxl
ModuleNotFoundError: No module named 'openpyxl'
~/Documents/github_pascalandy/localGPT|main⚡ ⇒

For the records, I was able to run privategpt smoothly.

M1/M2 Macbook users:

Something is not the same as the README on step 4

~/miniconda3|⇒  tree -L 1
.
├── LICENSE.txt
├── bin
├── conda-meta
├── condabin
├── envs
├── etc
├── include
├── lib
├── man
├── pkgs
├── python.app
├── share
├── shell
└── ssl

something is off about charset_normalizer

andy16@andy16:~/Documents/github_pascalandy/localGPT|main⚡ ⇒  pip uninstall charset_normalizer
Found existing installation: charset-normalizer 2.1.1
Uninstalling charset-normalizer-2.1.1:
  Would remove:
    /Users/andy16/miniconda3/bin/normalizer
    /Users/andy16/miniconda3/lib/python3.10/site-packages/charset_normalizer-2.1.1.dist-info/*
    /Users/andy16/miniconda3/lib/python3.10/site-packages/charset_normalizer/*
  Would not remove (might be manually added):
    /Users/andy16/miniconda3/lib/python3.10/site-packages/charset_normalizer/.DS_Store
Proceed (Y/n)? y
  Successfully uninstalled charset-normalizer-2.1.1
andy16@andy16:~/Documents/github_pascalandy/localGPT|main⚡ ⇒  pip install charset_normalizer
Collecting charset_normalizer
  Using cached charset_normalizer-3.1.0-cp310-cp310-macosx_11_0_arm64.whl (123 kB)
Installing collected packages: charset_normalizer
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
requests 2.28.1 requires charset-normalizer<3,>=2, but you have charset-normalizer 3.1.0 which is incompatible.

versions

python --version
Python 3.10.10

install looks ok

andy16@andy16:~/Documents/github_pascalandy/localGPT|main⚡ ⇒  pip install -r requirements.txt
Requirement already satisfied: langchain==0.0.166 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from -r requirements.txt (line 1)) (0.0.166)
Requirement already satisfied: chromadb==0.3.22 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from -r requirements.txt (line 2)) (0.3.22)
Requirement already satisfied: llama-cpp-python==0.1.48 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from -r requirements.txt (line 3)) (0.1.48)
Requirement already satisfied: urllib3==1.26.6 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from -r requirements.txt (line 4)) (1.26.6)
Requirement already satisfied: pdfminer.six==20221105 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from -r requirements.txt (line 5)) (20221105)
Requirement already satisfied: InstructorEmbedding in /Users/andy16/miniconda3/lib/python3.10/site-packages (from -r requirements.txt (line 6)) (1.0.1)
Requirement already satisfied: sentence-transformers in /Users/andy16/miniconda3/lib/python3.10/site-packages (from -r requirements.txt (line 7)) (2.2.2)
Requirement already satisfied: faiss-cpu in /Users/andy16/miniconda3/lib/python3.10/site-packages (from -r requirements.txt (line 8)) (1.7.4)
Requirement already satisfied: huggingface_hub in /Users/andy16/miniconda3/lib/python3.10/site-packages (from -r requirements.txt (line 9)) (0.15.1)
Requirement already satisfied: transformers in /Users/andy16/miniconda3/lib/python3.10/site-packages (from -r requirements.txt (line 10)) (4.29.2)
Requirement already satisfied: protobuf==3.20.0 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from -r requirements.txt (line 11)) (3.20.0)
Requirement already satisfied: accelerate in /Users/andy16/miniconda3/lib/python3.10/site-packages (from -r requirements.txt (line 12)) (0.19.0)
Requirement already satisfied: bitsandbytes in /Users/andy16/miniconda3/lib/python3.10/site-packages (from -r requirements.txt (line 13)) (0.39.0)
Requirement already satisfied: click in /Users/andy16/miniconda3/lib/python3.10/site-packages (from -r requirements.txt (line 14)) (8.1.3)
Requirement already satisfied: SQLAlchemy<3,>=1.4 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from langchain==0.0.166->-r requirements.txt (line 1)) (2.0.15)
Requirement already satisfied: dataclasses-json<0.6.0,>=0.5.7 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from langchain==0.0.166->-r requirements.txt (line 1)) (0.5.7)
Requirement already satisfied: async-timeout<5.0.0,>=4.0.0 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from langchain==0.0.166->-r requirements.txt (line 1)) (4.0.2)
Requirement already satisfied: numpy<2,>=1 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from langchain==0.0.166->-r requirements.txt (line 1)) (1.24.3)
Requirement already satisfied: pydantic<2,>=1 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from langchain==0.0.166->-r requirements.txt (line 1)) (1.10.8)
Requirement already satisfied: PyYAML>=5.4.1 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from langchain==0.0.166->-r requirements.txt (line 1)) (6.0)
Requirement already satisfied: numexpr<3.0.0,>=2.8.4 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from langchain==0.0.166->-r requirements.txt (line 1)) (2.8.4)
Requirement already satisfied: aiohttp<4.0.0,>=3.8.3 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from langchain==0.0.166->-r requirements.txt (line 1)) (3.8.4)
Requirement already satisfied: requests<3,>=2 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from langchain==0.0.166->-r requirements.txt (line 1)) (2.28.1)
Requirement already satisfied: tqdm>=4.48.0 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from langchain==0.0.166->-r requirements.txt (line 1)) (4.65.0)
Requirement already satisfied: openapi-schema-pydantic<2.0,>=1.2 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from langchain==0.0.166->-r requirements.txt (line 1)) (1.2.4)
Requirement already satisfied: tenacity<9.0.0,>=8.1.0 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from langchain==0.0.166->-r requirements.txt (line 1)) (8.2.2)
Requirement already satisfied: uvicorn[standard]>=0.18.3 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from chromadb==0.3.22->-r requirements.txt (line 2)) (0.22.0)
Requirement already satisfied: hnswlib>=0.7 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from chromadb==0.3.22->-r requirements.txt (line 2)) (0.7.0)
Requirement already satisfied: pandas>=1.3 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from chromadb==0.3.22->-r requirements.txt (line 2)) (2.0.2)
Requirement already satisfied: typing-extensions>=4.5.0 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from chromadb==0.3.22->-r requirements.txt (line 2)) (4.5.0)
Requirement already satisfied: posthog>=2.4.0 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from chromadb==0.3.22->-r requirements.txt (line 2)) (3.0.1)
Requirement already satisfied: fastapi>=0.85.1 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from chromadb==0.3.22->-r requirements.txt (line 2)) (0.96.0)
Requirement already satisfied: clickhouse-connect>=0.5.7 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from chromadb==0.3.22->-r requirements.txt (line 2)) (0.5.25)
Requirement already satisfied: duckdb>=0.7.1 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from chromadb==0.3.22->-r requirements.txt (line 2)) (0.8.0)
Requirement already satisfied: cryptography>=36.0.0 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from pdfminer.six==20221105->-r requirements.txt (line 5)) (39.0.1)
Requirement already satisfied: charset-normalizer>=2.0.0 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from pdfminer.six==20221105->-r requirements.txt (line 5)) (2.1.1)
Requirement already satisfied: scikit-learn in /Users/andy16/miniconda3/lib/python3.10/site-packages (from sentence-transformers->-r requirements.txt (line 7)) (1.2.2)
Requirement already satisfied: sentencepiece in /Users/andy16/miniconda3/lib/python3.10/site-packages (from sentence-transformers->-r requirements.txt (line 7)) (0.1.99)
Requirement already satisfied: torch>=1.6.0 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from sentence-transformers->-r requirements.txt (line 7)) (2.1.0.dev20230604)
Requirement already satisfied: scipy in /Users/andy16/miniconda3/lib/python3.10/site-packages (from sentence-transformers->-r requirements.txt (line 7)) (1.10.1)
Requirement already satisfied: nltk in /Users/andy16/miniconda3/lib/python3.10/site-packages (from sentence-transformers->-r requirements.txt (line 7)) (3.8.1)
Requirement already satisfied: torchvision in /Users/andy16/miniconda3/lib/python3.10/site-packages (from sentence-transformers->-r requirements.txt (line 7)) (0.13.1a0)
Requirement already satisfied: packaging>=20.9 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from huggingface_hub->-r requirements.txt (line 9)) (23.0)
Requirement already satisfied: fsspec in /Users/andy16/miniconda3/lib/python3.10/site-packages (from huggingface_hub->-r requirements.txt (line 9)) (2023.5.0)
Requirement already satisfied: filelock in /Users/andy16/miniconda3/lib/python3.10/site-packages (from huggingface_hub->-r requirements.txt (line 9)) (3.9.0)
Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from transformers->-r requirements.txt (line 10)) (0.13.3)
Requirement already satisfied: regex!=2019.12.17 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from transformers->-r requirements.txt (line 10)) (2023.6.3)
Requirement already satisfied: psutil in /Users/andy16/miniconda3/lib/python3.10/site-packages (from accelerate->-r requirements.txt (line 12)) (5.9.5)
Requirement already satisfied: attrs>=17.3.0 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.166->-r requirements.txt (line 1)) (23.1.0)
Requirement already satisfied: aiosignal>=1.1.2 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.166->-r requirements.txt (line 1)) (1.3.1)
Requirement already satisfied: multidict<7.0,>=4.5 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.166->-r requirements.txt (line 1)) (6.0.4)
Requirement already satisfied: yarl<2.0,>=1.0 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.166->-r requirements.txt (line 1)) (1.9.2)
Requirement already satisfied: frozenlist>=1.1.1 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.166->-r requirements.txt (line 1)) (1.3.3)
Requirement already satisfied: certifi in /Users/andy16/miniconda3/lib/python3.10/site-packages (from clickhouse-connect>=0.5.7->chromadb==0.3.22->-r requirements.txt (line 2)) (2023.5.7)
Requirement already satisfied: zstandard in /Users/andy16/miniconda3/lib/python3.10/site-packages (from clickhouse-connect>=0.5.7->chromadb==0.3.22->-r requirements.txt (line 2)) (0.19.0)
Requirement already satisfied: lz4 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from clickhouse-connect>=0.5.7->chromadb==0.3.22->-r requirements.txt (line 2)) (4.3.2)
Requirement already satisfied: pytz in /Users/andy16/miniconda3/lib/python3.10/site-packages (from clickhouse-connect>=0.5.7->chromadb==0.3.22->-r requirements.txt (line 2)) (2023.3)
Requirement already satisfied: cffi>=1.12 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from cryptography>=36.0.0->pdfminer.six==20221105->-r requirements.txt (line 5)) (1.15.1)
Requirement already satisfied: marshmallow<4.0.0,>=3.3.0 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from dataclasses-json<0.6.0,>=0.5.7->langchain==0.0.166->-r requirements.txt (line 1)) (3.19.0)
Requirement already satisfied: typing-inspect>=0.4.0 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from dataclasses-json<0.6.0,>=0.5.7->langchain==0.0.166->-r requirements.txt (line 1)) (0.9.0)
Requirement already satisfied: marshmallow-enum<2.0.0,>=1.5.1 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from dataclasses-json<0.6.0,>=0.5.7->langchain==0.0.166->-r requirements.txt (line 1)) (1.5.1)
Requirement already satisfied: starlette<0.28.0,>=0.27.0 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from fastapi>=0.85.1->chromadb==0.3.22->-r requirements.txt (line 2)) (0.27.0)
Requirement already satisfied: tzdata>=2022.1 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from pandas>=1.3->chromadb==0.3.22->-r requirements.txt (line 2)) (2023.3)
Requirement already satisfied: python-dateutil>=2.8.2 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from pandas>=1.3->chromadb==0.3.22->-r requirements.txt (line 2)) (2.8.2)
Requirement already satisfied: monotonic>=1.5 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from posthog>=2.4.0->chromadb==0.3.22->-r requirements.txt (line 2)) (1.6)
Requirement already satisfied: six>=1.5 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from posthog>=2.4.0->chromadb==0.3.22->-r requirements.txt (line 2)) (1.16.0)
Requirement already satisfied: backoff>=1.10.0 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from posthog>=2.4.0->chromadb==0.3.22->-r requirements.txt (line 2)) (2.2.1)
Requirement already satisfied: idna<4,>=2.5 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from requests<3,>=2->langchain==0.0.166->-r requirements.txt (line 1)) (3.4)
Requirement already satisfied: sympy in /Users/andy16/miniconda3/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers->-r requirements.txt (line 7)) (1.11.1)
Requirement already satisfied: networkx in /Users/andy16/miniconda3/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers->-r requirements.txt (line 7)) (2.8.4)
Requirement already satisfied: jinja2 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers->-r requirements.txt (line 7)) (3.1.2)
Requirement already satisfied: h11>=0.8 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.3->chromadb==0.3.22->-r requirements.txt (line 2)) (0.14.0)
Requirement already satisfied: websockets>=10.4 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.3->chromadb==0.3.22->-r requirements.txt (line 2)) (11.0.3)
Requirement already satisfied: httptools>=0.5.0 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.3->chromadb==0.3.22->-r requirements.txt (line 2)) (0.5.0)
Requirement already satisfied: python-dotenv>=0.13 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.3->chromadb==0.3.22->-r requirements.txt (line 2)) (1.0.0)
Requirement already satisfied: watchfiles>=0.13 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.3->chromadb==0.3.22->-r requirements.txt (line 2)) (0.19.0)
Requirement already satisfied: uvloop!=0.15.0,!=0.15.1,>=0.14.0 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.3->chromadb==0.3.22->-r requirements.txt (line 2)) (0.17.0)
Requirement already satisfied: joblib in /Users/andy16/miniconda3/lib/python3.10/site-packages (from nltk->sentence-transformers->-r requirements.txt (line 7)) (1.2.0)
Requirement already satisfied: threadpoolctl>=2.0.0 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from scikit-learn->sentence-transformers->-r requirements.txt (line 7)) (3.1.0)
Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from torchvision->sentence-transformers->-r requirements.txt (line 7)) (9.4.0)
Requirement already satisfied: pycparser in /Users/andy16/miniconda3/lib/python3.10/site-packages (from cffi>=1.12->cryptography>=36.0.0->pdfminer.six==20221105->-r requirements.txt (line 5)) (2.21)
Requirement already satisfied: anyio<5,>=3.4.0 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from starlette<0.28.0,>=0.27.0->fastapi>=0.85.1->chromadb==0.3.22->-r requirements.txt (line 2)) (3.7.0)
Requirement already satisfied: mypy-extensions>=0.3.0 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from typing-inspect>=0.4.0->dataclasses-json<0.6.0,>=0.5.7->langchain==0.0.166->-r requirements.txt (line 1)) (1.0.0)
Requirement already satisfied: MarkupSafe>=2.0 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from jinja2->torch>=1.6.0->sentence-transformers->-r requirements.txt (line 7)) (2.1.1)
Requirement already satisfied: mpmath>=0.19 in /Users/andy16/miniconda3/lib/python3.10/site-packages/mpmath-1.2.1-py3.10.egg (from sympy->torch>=1.6.0->sentence-transformers->-r requirements.txt (line 7)) (1.2.1)
Requirement already satisfied: exceptiongroup in /Users/andy16/miniconda3/lib/python3.10/site-packages (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi>=0.85.1->chromadb==0.3.22->-r requirements.txt (line 2)) (1.1.1)
Requirement already satisfied: sniffio>=1.1 in /Users/andy16/miniconda3/lib/python3.10/site-packages (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi>=0.85.1->chromadb==0.3.22->-r requirements.txt (line 2)) (1.3.0)
PromtEngineer commented 1 year ago

Updating to the latest version will most probably fix this:

pip install --force-reinstall charset-normalizer==3.1.0

ljmatan commented 11 months ago

Updating to the latest version will most probably fix this:

Didn't help.

What helped was looking at the error messages which suggested I should do:

$ python3
import ssl
import ntlk

try:
    _create_unverified_https_context = ssl._create_unverified_context
except AttributeError:
    pass
else:
    ssl._create_default_https_context = _create_unverified_https_context

nltk.download('punkt')
nltk.download('averaged_perceptron_tagger')