search

langgenius / dify

Dify is an open-source LLM app development platform. Dify's intuitive interface combines AI workflow, RAG pipeline, agent capabilities, model management, observability features and more, letting you quickly go from prototype to production.
https://dify.ai
Other
43.31k stars 6k forks source link

Failed to call xinference and azure openai text embedding models #4736

Closed Starlento closed 3 months ago

Starlento commented 3 months ago

Self Checks

Dify version

0.6.9

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Add xinference text embedding model. Drag a pdf to knowledge base.

I also tried azure openai service text-embedding. Failed as well, but the gpt-4o API works. Logically since I add the url successfully, the service should be valid.

image

✔️ Expected Behavior

successfully call the api and create vector db.

❌ Actual Behavior

image image


[2024-05-28 12:28:46,829: ERROR/MainProcess] consume document failed
Traceback (most recent call last):
  File "/app/api/core/indexing_runner.py", line 73, in run
    self._load(
  File "/app/api/core/indexing_runner.py", line 678, in _load
    tokens += future.result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/app/api/core/indexing_runner.py", line 733, in _process_chunk
    index_processor.load(dataset, chunk_documents, with_keywords=False)
  File "/app/api/core/rag/index_processor/processor/paragraph_index_processor.py", line 59, in load
    vector = Vector(dataset)
  File "/app/api/core/rag/datasource/vdb/vector_factory.py", line 23, in __init__
    self._vector_processor = self._init_vector()
  File "/app/api/core/rag/datasource/vdb/vector_factory.py", line 48, in _init_vector
    return WeaviateVector(
  File "/app/api/core/rag/datasource/vdb/weaviate/weaviate_vector.py", line 31, in __init__
    self._client = self._init_client(config)
  File "/app/api/core/rag/datasource/vdb/weaviate/weaviate_vector.py", line 40, in _init_client
    client = weaviate.Client(
  File "/usr/local/lib/python3.10/site-packages/weaviate/client.py", line 150, in __init__
    self._connection = Connection(
  File "/usr/local/lib/python3.10/site-packages/weaviate/connect/connection.py", line 654, in __init__
    self._server_version = self.get_meta()["version"]
  File "/usr/local/lib/python3.10/site-packages/weaviate/connect/connection.py", line 678, in get_meta
    raise UnexpectedStatusCodeException("Meta endpoint", response)
weaviate.exceptions.UnexpectedStatusCodeException: Meta endpoint! Unexpected status code: 503, with response body: None.
dosubot[bot] commented 3 months ago

To resolve the "503 Service Unavailable" error when calling xinference and Azure OpenAI text embedding models in your self-hosted (Docker) setup of Dify version 0.6.9, follow these steps:

  1. Check Service Dependencies: Ensure all dependent services (like db, redis, weaviate, etc.) are up and running. The depends_on directive in your docker-compose.yaml should help manage this.

  2. Network Configuration: Ensure the services are correctly networked. The ssrf_proxy_network and default networks should be properly configured.

  3. Service URLs and Ports: Verify that the service URLs and ports are correctly set and accessible. Ensure that the WEAVIATE_ENDPOINT and other service endpoints are reachable from the api and worker services.

  4. Logs and Debugging: Check the logs of the services to identify any specific issues. Enable more detailed logging by setting LOG_LEVEL to DEBUG.

  5. Validate Azure OpenAI Credentials: Ensure the credentials for Azure OpenAI are correctly set. The AzureOpenAITextEmbeddingModel class in Dify's codebase validates credentials and requires openai_api_base, openai_api_key, openai_api_version, and base_model_name. Make sure these are correctly configured in your environment variables.

  6. Xinference Configuration: Ensure Xinference is correctly deployed and accessible. Follow these steps to deploy and configure Xinference:

    • Install Xinference: 
      $ pip install "xinference[all]"
    • Start Xinference Locally: 
      $ xinference-local
    • Network Configuration for Docker: Ensure the Dify container can access the Xinference endpoint. Use the host IP address instead of localhost.
    • Deploy the Model: Visit http://127.0.0.1:9997, select and deploy the model you need.
    • Obtain the Model UID: Copy the model ID from the Running Models page.
    • Connect the Model in Dify: In Settings > Model Providers > Xinference, enter the model details and save.

Here is a snippet of the docker-compose.yaml that shows the configuration of the services:

version: '3'
services:
  api:
    image: langgenius/dify-api:0.6.9
    restart: always
    environment:
      MODE: api
      LOG_LEVEL: INFO
      SECRET_KEY: sk-9f73s3ljTXVcMT3Blb3ljTqtsKiGHXVcMT3BlbkFJLK7U
      DB_HOST: db
      DB_PORT: 5432
      DB_DATABASE: dify
      REDIS_HOST: redis
      REDIS_PORT: 6379
      VECTOR_STORE: weaviate
      WEAVIATE_ENDPOINT: http://weaviate:8080
      WEAVIATE_API_KEY: WVF5YThaHlkYwhGUSmCRgsX3tD5ngdN8pkih
    depends_on:
      - db
      - redis
    networks:
      - ssrf_proxy_network
      - default

  worker:
    image: langgenius/dify-api:0.6.9
    restart: always
    environment:
      MODE: worker
      LOG_LEVEL: INFO
      SECRET_KEY: sk-9f73s3ljTXVcMT3Blb3ljTqtsKiGHXVcMT3BlbkFJLK7U
      DB_HOST: db
      DB_PORT: 5432
      DB_DATABASE: dify
      REDIS_HOST: redis
      REDIS_PORT: 6379
      VECTOR_STORE: weaviate
      WEAVIATE_ENDPOINT: http://weaviate:8080
      WEAVIATE_API_KEY: WVF5YThaHlkYwhGUSmCRgsX3tD5ngdN8pkih
    depends_on:
      - db
      - redis
    networks:
      - ssrf_proxy_network
      - default

  db:
    image: postgres:15-alpine
    restart: always
    environment:
      POSTGRES_PASSWORD: difyai123456
      POSTGRES_DB: dify
    volumes:
      - ./volumes/db/data:/var/lib/postgresql/data
    healthcheck:
      test: [ "CMD", "pg_isready" ]
      interval: 1s
      timeout: 3s
      retries: 30

  redis:
    image: redis:6-alpine
    restart: always
    volumes:
      - ./volumes/redis/data:/data
    command: redis-server --requirepass difyai123456
    healthcheck:
      test: [ "CMD", "redis-cli", "ping" ]

  weaviate:
    image: semitechnologies/weaviate:1.19.0
    restart: always
    volumes:
      - ./volumes/weaviate:/var/lib/weaviate
    environment:
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_APIKEY_ENABLED: 'true'
      AUTHENTICATION_APIKEY_ALLOWED_KEYS: 'WVF5YThaHlkYwhGUSmCRgsX3tD5ngdN8pkih'
    networks:
      - ssrf_proxy_network
      - default

networks:
  ssrf_proxy_network:
    driver: bridge
    internal: true
  default:
    driver: bridge

Ensure that the XinferenceTextEmbeddingModel class is correctly implemented and that the credentials are validated properly. If the issue persists, check the logs for any specific errors and adjust the configurations accordingly [1][2][3][4][5].

#### Details **About Dosu** This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

Starlento commented 3 months ago

Solved by: https://github.com/langgenius/dify/issues/734#issuecomment-1925580564