search

deepset-ai / haystack

:mag: AI orchestration framework to build customizable, production-ready LLM applications. Connect components (models, vector DBs, file converters) to pipelines or agents that can interact with your data. With advanced retrieval methods, it's best suited for building RAG, question answering, semantic search or conversational agent chatbots.
https://haystack.deepset.ai
Apache License 2.0
17.62k stars 1.91k forks source link

Docker demo: `Failed loading pipeline component 'Reader': Unable to load weights from pytorch checkpoint file` causes `Connection error` #1357

Closed sriprad closed 3 years ago

sriprad commented 3 years ago

Describe the bug I am trying to replicate the docker (demo) with the webapp (streamlit) version of the code. When i run docker compose up i am getting the below error

Error message Error : raise ConnectionError(e, request=request) ui_1 | requests.exceptions.ConnectionError: HTTPConnectionPool(host='haystack-api', port=8000): Max retries exceeded with url: /query (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f5cce1a2350>: Failed to establish a new connection: [Errno 111] Connection refused'))

image

System:

tholor commented 3 years ago

As mentioned in #1350, please share:

Otherwise it will be impossible to debug / replicate.

JH95-ai commented 3 years ago

I have the same problems.Following is my logs. F:\github_download\semantic_search\haystack>docker-compose pull [+] Running 13/22

F:\github_download\semantic_search\haystack>docker-compose up [+] Running 3/3

JH95-ai commented 3 years ago

docker stats log is following: CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS 61f306700043 haystack_haystack-api_1 0.05% 282MiB / 12.45GiB 2.21% 2.59MB / 95.3kB 0B / 0B 2 5cd1972a99f4 haystack_elasticsearch_1 0.26% 1.298GiB / 12.45GiB 10.43% 5.84kB / 3.71kB 0B / 0B 61 d7934023abd3 haystack_ui_1 2.57% 101.8MiB / 12.45GiB 0.80% 142kB / 203kB 0B / 0B 7

brandenchan commented 3 years ago

Hi @NeekHua , there are two things that catch my eye in your logs

streamlit.caching.CacheKeyNotFoundError: Key not found in mem cache
Exception: Failed loading pipeline component 'Reader': Unable to load weights from pytorch checkpoint file for 'deepset/roberta-base-squad2' at '/root/.cache/huggingface/transformers/eac3273a8097dda671e3bea1db32c616e74f36a306c65b4858171c98d6db83e9.084aa7284f3a51fa1c8f0641aa04c47d366fbd18711f29d0a995693cfdbc9c9e'If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True.

Is it possible that the problem lies in your cache? Perhaps something around permissions?

One way to test this would be to run a script that loads a huggingface model from cache. In Haystack, you could try running a script like this twice

reader = FARMReader(model_name_or_path="deepset/roberta-base-squad2")
yingzwang commented 3 years ago

I've encountered the same issue. Have you figured out a solution? @NeekHua

XingYiBao commented 3 years ago

@julian-risch i encountered the same /similar issue when running docker-compose up. is it possible you seperate the components into several docker image so that we can run them one by one? Thanks,

julian-risch commented 3 years ago

I am not sure whether I correctly understand your idea. In https://github.com/deepset-ai/haystack/blob/master/docker-compose.yml we already have separate components with three images: "deepset/haystack-cpu:latest", "deepset/elasticsearch-game-of-thrones", and "deepset/haystack-streamlit-ui:latest"

XingYiBao commented 3 years ago

maybe you can ignore my comments. @julian-risch however, I did encounter the following error message when run with sudo docker-compose up, please kindly advise how to handle it. Thanks a lot!

haystack-api_1 | During handling of the above exception, another exception occurred: haystack-api_1 | haystack-api_1 | Traceback (most recent call last): haystack-api_1 | File "/usr/local/lib/python3.7/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker haystack-api_1 | worker.init_process() haystack-api_1 | File "/usr/local/lib/python3.7/site-packages/uvicorn/workers.py", line 66, in init_process haystack-api_1 | super(UvicornWorker, self).init_process() haystack-api_1 | File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 134, in init_process haystack-api_1 | self.load_wsgi() haystack-api_1 | File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi haystack-api_1 | self.wsgi = self.app.wsgi() haystack-api_1 | File "/usr/local/lib/python3.7/site-packages/gunicorn/app/base.py", line 67, in wsgi haystack-api_1 | self.callable = self.load() haystack-api_1 | File "/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 58, in load haystack-api_1 | return self.load_wsgiapp() haystack-api_1 | File "/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp haystack-api_1 | return util.import_app(self.app_uri) haystack-api_1 | File "/usr/local/lib/python3.7/site-packages/gunicorn/util.py", line 359, in import_app haystack-api_1 | mod = importlib.import_module(module) haystack-api_1 | File "/usr/local/lib/python3.7/importlib/init.py", line 127, in import_module haystack-api_1 | return _bootstrap._gcd_import(name[level:], package, level) haystack-api_1 | File "", line 1006, in _gcd_import haystack-api_1 | File "", line 983, in _find_and_load haystack-api_1 | File "", line 967, in _find_and_load_unlocked haystack-api_1 | File "", line 677, in _load_unlocked haystack-api_1 | File "", line 728, in exec_module haystack-api_1 | File "", line 219, in _call_with_frames_removed haystack-api_1 | File "/home/user/rest_api/application.py", line 8, in haystack-api_1 | from rest_api.controller.router import router as api_router haystack-api_1 | File "/home/user/rest_api/controller/router.py", line 3, in haystack-api_1 | from rest_api.controller import file_upload, search, feedback haystack-api_1 | File "/home/user/rest_api/controller/search.py", line 44, in haystack-api_1 | PIPELINE = Pipeline.load_from_yaml(Path(PIPELINE_YAML_PATH), pipeline_name=QUERY_PIPELINE_NAME) haystack-api_1 | File "/home/user/haystack/pipeline.py", line 430, in load_from_yaml haystack-api_1 | component = cls._load_or_get_component(name=name, definitions=definitions, components=components) haystack-api_1 | File "/home/user/haystack/pipeline.py", line 463, in _load_or_get_component haystack-api_1 | raise Exception(f"Failed loading pipeline component '{name}': {e}") haystack-api_1 | Exception: Failed loading pipeline component 'Reader': Unable to load weights from pytorch checkpoint file for 'deepset/roberta-base-squad2' at '/root/.cache/huggingface/transformers/eac3273a8097dda671e3bea1db32c616e74f36a306c65b4858171c98d6db83e9.084aa7284f3a51fa1c8f0641aa04c47d366fbd18711f29d0a995693cfdbc9c9e'If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True. haystack-api_1 | [2021-09-21 15:07:36 +0000] [13] [INFO] Worker exiting (pid: 13) haystack-api_1 | [2021-09-21 15:07:39 +0000] [1] [INFO] Shutting down: Master haystack-api_1 | [2021-09-21 15:07:39 +0000] [1] [INFO] Reason: Worker failed to boot. haystack_haystack-api_1 exited with code 3

ZanSara commented 3 years ago

Hi @XingYiBao , I can't replicate your issue with the information you provided. What I can suggest is to run docker without sudo, and if it still doesn't work, to give us some more information about your setup, like:

and any other information that might help.

Note: the root of the issue (Unable to load weights from pytorch checkpoint file for 'deepset/roberta-base-squad2' at '/root/.cache/huggingface/transformers) seems to come from transformers (https://github.com/huggingface/transformers), and it's an odd one: https://github.com/huggingface/transformers/blob/51ee20fc26381ca8aba4d4da9b410379302ee1d1/src/transformers/modeling_utils.py#L1361. If somebody can provide a way to reproduce this issue reliably I will definitely investigate.

XingYiBao commented 3 years ago

@ZanSara seems it is the default docker-compose.yml issue. Made some change and now it works. anyway, thanks a lot!

ZanSara commented 3 years ago

Glad to hear! If you could share the changes you've made to the docker-compose.yml we can fix the default one, so that it works for everybody :slightly_smiling_face:

XingYiBao commented 3 years ago

@ZanSara no problem. Below are some changes we made to the docker-compose.yml file. To use the network_mode:bridge, otherwise the default ip created by docker MAY conflict with your internal network ip addresses. The sample file after changing is as below:

version: "3"
services:
  haystack-api:
    build:
      context: .
      dockerfile: Dockerfile
    image: "/deepset/haystack-cpu:latest"
    # Mount custom Pipeline YAML and custom Components.
    # volumes:
    #   - ./rest_api/pipeline:/home/user/rest_api/pipeline
    ports:
      - 8000:8000
    environment:
      # See rest_api/pipelines.yaml for configurations of Search & Indexing Pipeline.
      #- ELASTICSEARCHDOCUMENTSTORE_PARAMS_HOST=elasticsearch
      #- DOCUMENTSTORE_PARAMS_HOST=elasticsearch
      - DOCUMENTSTORE_PARAMS_HOST={ip address of elasticsearch host}
    restart: always
    #depends_on:
    #  - elasticsearch
    command: "/bin/bash -c 'sleep 15 && gunicorn rest_api.application:app -b 0.0.0.0 -k uvicorn.workers.UvicornWorker --workers 1 --timeout 180'"
    network_mode: "bridge"
  elasticsearch:
    # This will start an empty elasticsearch instance (so you have to add your documents yourself)
    #image: "elasticsearch:7.9.2"
    # If you want a demo image instead that is "ready-to-query" with some indexed Game of Thrones articles:
    image: "/deepset/elasticsearch-game-of-thrones"
    ports:
      - 9200:9200
    environment:
      - discovery.type=single-node
    network_mode: "bridge"
  ui:
    build:
      context: ui
      dockerfile: Dockerfile
    image: "/deepset/haystack-streamlit-ui:latest"
    ports:
      - 8501:8501
    environment:
      - API_ENDPOINT=http://{your ip address}:8000
      - EVAL_FILE=eval_labels_example.csv
    network_mode: "bridge"

BTW, another SEPERATE issue: I strongly recommend that you specify the version for each docker image so that it won't use the latest one, because the latest one may result in unexpected problem. For example, this is the docker-compose-gpu.yml file after I changed your original one:

version: "3"
services:
  haystack-api:
    build:
      context: .
      dockerfile: Dockerfile
    image: "/deepset/haystack-gpu:0.10.0"

while the docker-all.repo.ebaotech.com/deepset/haystack-gpu:latest does NOT work.

ZanSara commented 3 years ago

Fantastic, thank you! (I took the liberty of adding a code block around the YAML for readability, I hope you don't mind).

Yes, the latest image might break sometimes, so pinning it to the latest release it's a good idea. I'm going to discuss it with the rest of the team.

Closing for now. Is somebody else have the same issue, feel free to re-open or to just open a new issue :slightly_smiling_face: