怎么能对接dify，应该修改哪里呢？

thirttyyy commented 1 year ago

Hi，我想知道修改哪里可以用本地llm模型对接上dify，我使用如下修改：

os.environ["OPENAI_API_BASE"] = "http://xxx"
os.environ["OPENAI_API_KEY"] = "xxx"

但是进入dify后会要求验证我的Open AI APIkey

xusenlinzy commented 1 year ago

@thirttyyy 试一下取消API KEY的验证

os.environ["DISABLE_PROVIDER_CONFIG_VALIDATION"] = True

进去之后随意输入一个API KEY，保存之后应该就可以使用了

thirttyyy commented 1 year ago

感谢您的回复，试了下还是会去请求openai验证秘钥，是不是我写的位置不对。我是用 dify 推荐的 dify/docker下docker-compose up -d命令启动，按照我的理解，命令会根据api，web，nginx等启动几个容器，所以我把环境命令：
os.environ["DISABLE_PROVIDER_CONFIG_VALIDATION"] = True
os.environ["OPENAI_API_BASE"] = "http://xxx"
os.environ["OPENAI_API_KEY"] = "xxx"
写在了dify/api/app.py文件中，是不是没起到作用？

xusenlinzy commented 1 year ago

最好不要修改app.py文件，通过在docker-compose.yml文件中添加环境变量

version: '3.1'
services:
  # API service
  api:
    image: langgenius/dify-api:0.3.0
    restart: always
    environment:
      OPENAI_API_BASE: http://xxx.xxx.x.xx:80/v1
      DISABLE_PROVIDER_CONFIG_VALIDATION: 'true'
      # Startup mode, 'api' starts the API server.
      MODE: api
      # The log level for the application. Supported values are `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`
      LOG_LEVEL: INFO
      # A secret key that is used for securely signing the session cookie and encrypting sensitive information on the database. You can generate a strong key using `openssl rand -base64 42`.
      SECRET_KEY: sk-9f73s3ljTXVcMT3Blb3ljTqtsKiGHXVcMT3BlbkFJLK7U
      # The base URL of console application, refers to the Console base URL of WEB service if console domain is
      # different from api or web app domain.
      # example: http://cloud.dify.ai
      CONSOLE_URL: ''
      # The URL for Service API endpoints，refers to the base URL of the current API service if api domain is
      # different from console domain.
      # example: http://api.dify.ai
      API_URL: ''
      # The URL for Web APP, refers to the Web App base URL of WEB service if web app domain is different from
      # console or api domain.
      # example: http://udify.app
      APP_URL: ''
      # When enabled, migrations will be executed prior to application startup and the application will start after the migrations have completed.
      MIGRATION_ENABLED: 'true'
      # The configurations of postgres database connection.
      # It is consistent with the configuration in the 'db' service below.
      DB_USERNAME: postgres
      DB_PASSWORD: difyai123456
      DB_HOST: db
      DB_PORT: 5432
      DB_DATABASE: dify
      # The configurations of redis connection.
      # It is consistent with the configuration in the 'redis' service below.
      REDIS_HOST: redis
      REDIS_PORT: 6379
      REDIS_USERNAME: ''
      REDIS_PASSWORD: difyai123456
      REDIS_USE_SSL: 'false'
      # use redis db 0 for redis cache
      REDIS_DB: 0
      # The configurations of session, Supported values are `sqlalchemy`. `redis`
      SESSION_TYPE: redis
      SESSION_REDIS_HOST: redis
      SESSION_REDIS_PORT: 6379
      SESSION_REDIS_USERNAME: ''
      SESSION_REDIS_PASSWORD: difyai123456
      SESSION_REDIS_USE_SSL: 'false'
      # use redis db 2 for session store
      SESSION_REDIS_DB: 2
      # The configurations of celery broker.
      # Use redis as the broker, and redis db 1 for celery broker.
      CELERY_BROKER_URL: redis://:difyai123456@redis:6379/1
      # Specifies the allowed origins for cross-origin requests to the Web API, e.g. https://dify.app or * for all origins.
      WEB_API_CORS_ALLOW_ORIGINS: '*'
      # Specifies the allowed origins for cross-origin requests to the console API, e.g. https://cloud.dify.ai or * for all origins.
      CONSOLE_CORS_ALLOW_ORIGINS: '*'
      # CSRF Cookie settings
      # Controls whether a cookie is sent with cross-site requests,
      # providing some protection against cross-site request forgery attacks
      #
      # Default: `SameSite=Lax, Secure=false, HttpOnly=true`
      # This default configuration supports same-origin requests using either HTTP or HTTPS,
      # but does not support cross-origin requests. It is suitable for local debugging purposes.
      #
      # If you want to enable cross-origin support,
      # you must use the HTTPS protocol and set the configuration to `SameSite=None, Secure=true, HttpOnly=true`.
      #
      # For **production** purposes, please set `SameSite=Lax, Secure=true, HttpOnly=true`.
      COOKIE_HTTPONLY: 'true'
      COOKIE_SAMESITE: 'Lax'
      COOKIE_SECURE: 'false'
      # The type of storage to use for storing user files. Supported values are `local` and `s3`, Default: `local`
      STORAGE_TYPE: local
      # The path to the local storage directory, the directory relative the root path of API service codes or absolute path. Default: `storage` or `/home/john/storage`.
      # only available when STORAGE_TYPE is `local`.
      STORAGE_LOCAL_PATH: storage
      # The S3 storage configurations, only available when STORAGE_TYPE is `s3`.
      S3_ENDPOINT: 'https://xxx.r2.cloudflarestorage.com'
      S3_BUCKET_NAME: 'difyai'
      S3_ACCESS_KEY: 'ak-difyai'
      S3_SECRET_KEY: 'sk-difyai'
      S3_REGION: 'us-east-1'
      # The type of vector store to use. Supported values are `weaviate`, `qdrant`.
      VECTOR_STORE: weaviate
      # The Weaviate endpoint URL. Only available when VECTOR_STORE is `weaviate`.
      WEAVIATE_ENDPOINT: http://weaviate:8080
      # The Weaviate API key.
      WEAVIATE_API_KEY: WVF5YThaHlkYwhGUSmCRgsX3tD5ngdN8pkih
      # The Qdrant endpoint URL. Only available when VECTOR_STORE is `qdrant`.
      QDRANT_URL: 'https://your-qdrant-cluster-url.qdrant.tech/'
      # The Qdrant API key.
      QDRANT_API_KEY: 'ak-difyai'
      # The DSN for Sentry error reporting. If not set, Sentry error reporting will be disabled.
      SENTRY_DSN: ''
      # The sample rate for Sentry events. Default: `1.0`
      SENTRY_TRACES_SAMPLE_RATE: 1.0
      # The sample rate for Sentry profiles. Default: `1.0`
      SENTRY_PROFILES_SAMPLE_RATE: 1.0
    depends_on:
      - db
      - redis
      - weaviate
    volumes:
      # Mount the storage directory to the container, for storing user files.
      - ./volumes/app/storage:/app/api/storage

  # worker service
  # The Celery worker for processing the queue.
  worker:
    image: langgenius/dify-api:0.3.0
    restart: always
    environment:
      OPENAI_API_BASE: http://xxx.xxx.x.xx:80/v1
      DISABLE_PROVIDER_CONFIG_VALIDATION: 'true'
      # Startup mode, 'worker' starts the Celery worker for processing the queue.
      MODE: worker

      # --- All the configurations below are the same as those in the 'api' service. ---

      # The log level for the application. Supported values are `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`
      LOG_LEVEL: INFO
      # A secret key that is used for securely signing the session cookie and encrypting sensitive information on the database. You can generate a strong key using `openssl rand -base64 42`.
      # same as the API service
      SECRET_KEY: sk-9f73s3ljTXVcMT3Blb3ljTqtsKiGHXVcMT3BlbkFJLK7U
      # The configurations of postgres database connection.
      # It is consistent with the configuration in the 'db' service below.
      DB_USERNAME: postgres
      DB_PASSWORD: difyai123456
      DB_HOST: db
      DB_PORT: 5432
      DB_DATABASE: dify
      # The configurations of redis cache connection.
      REDIS_HOST: redis
      REDIS_PORT: 6379
      REDIS_USERNAME: ''
      REDIS_PASSWORD: difyai123456
      REDIS_DB: 0
      REDIS_USE_SSL: 'false'
      # The configurations of celery broker.
      CELERY_BROKER_URL: redis://:difyai123456@redis:6379/1
      # The type of storage to use for storing user files. Supported values are `local` and `s3`, Default: `local`
      STORAGE_TYPE: local
      STORAGE_LOCAL_PATH: storage
      # The Vector store configurations.
      VECTOR_STORE: weaviate
      WEAVIATE_ENDPOINT: http://weaviate:8080
      WEAVIATE_API_KEY: WVF5YThaHlkYwhGUSmCRgsX3tD5ngdN8pkih
    depends_on:
      - db
      - redis
      - weaviate
    volumes:
      # Mount the storage directory to the container, for storing user files.
      - ./volumes/app/storage:/app/api/storage

  # Frontend web application.
  web:
    image: langgenius/dify-web:0.3.0
    restart: always
    environment:
      EDITION: SELF_HOSTED
      # The base URL of console application, refers to the Console base URL of WEB service if console domain is
      # different from api or web app domain.
      # example: http://cloud.dify.ai
      CONSOLE_URL: ''
      # The URL for Web APP, refers to the Web App base URL of WEB service if web app domain is different from
      # console or api domain.
      # example: http://udify.app
      APP_URL: ''

  # The postgres database.
  db:
    image: postgres:15-alpine
    restart: always
    environment:
      # The password for the default postgres user.
      POSTGRES_PASSWORD: difyai123456
      # The name of the default postgres database.
      POSTGRES_DB: dify
      # postgres data directory
      PGDATA: /var/lib/postgresql/data/pgdata
    volumes:
      - ./volumes/db/data:/var/lib/postgresql/data
    ports:
      - "5432:5432"

  # The redis cache.
  redis:
    image: redis:6-alpine
    restart: always
    volumes:
      # Mount the redis data directory to the container.
      - ./volumes/redis/data:/data
    # Set the redis password when startup redis server.
    command: redis-server --requirepass difyai123456

  # The Weaviate vector store.
  weaviate:
    image: semitechnologies/weaviate:1.18.4
    restart: always
    volumes:
      # Mount the Weaviate data directory to the container.
      - ./volumes/weaviate:/var/lib/weaviate
    environment:
      # The Weaviate configurations
      # You can refer to the [Weaviate](https://weaviate.io/developers/weaviate/config-refs/env-vars) documentation for more information.
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'false'
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
      DEFAULT_VECTORIZER_MODULE: 'none'
      CLUSTER_HOSTNAME: 'node1'
      AUTHENTICATION_APIKEY_ENABLED: 'true'
      AUTHENTICATION_APIKEY_ALLOWED_KEYS: 'WVF5YThaHlkYwhGUSmCRgsX3tD5ngdN8pkih'
      AUTHENTICATION_APIKEY_USERS: 'hello@dify.ai'
      AUTHORIZATION_ADMINLIST_ENABLED: 'true'
      AUTHORIZATION_ADMINLIST_USERS: 'hello@dify.ai'

  # The nginx reverse proxy.
  # used for reverse proxying the API service and Web service.
  nginx:
    image: nginx:latest
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf
      - ./nginx/proxy.conf:/etc/nginx/proxy.conf
      - ./nginx/conf.d:/etc/nginx/conf.d
    depends_on:
      - api
      - web
    ports:
      - "80:80"

在api和worker中添加

OPENAI_API_BASE: http://xxx.xxx.x.xx:80/v1
DISABLE_PROVIDER_CONFIG_VALIDATION: 'true'

thirttyyy commented 1 year ago

绕过秘钥验证了，这样是对的，应该直接在docker-compose.yml上修改变量，我将关闭这个issue。

panmyuan commented 1 year ago

你好，我按照要求修改，但是还是会有问题，在linux上， DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): openaipublic.blob.core.windows.net:443 ERROR:root:Unknown Error in completion Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn conn = connection.create_connection( File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 72, in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): File "/usr/local/lib/python3.10/site-packages/gevent/_socketcommon.py", line 247, in getaddrinfo addrlist = get_hub().resolver.getaddrinfo(host, port, family, type, proto, flags) File "/usr/local/lib/python3.10/site-packages/gevent/resolver/thread.py", line 63, in getaddrinfo return self.pool.apply(_socket.getaddrinfo, args, kwargs) File "/usr/local/lib/python3.10/site-packages/gevent/pool.py", line 161, in apply return self.spawn(func, *args, *kwds).get() File "src/gevent/event.py", line 329, in gevent._gevent_cevent.AsyncResult.get File "src/gevent/event.py", line 359, in gevent._gevent_cevent.AsyncResult.get File "src/gevent/event.py", line 347, in gevent._gevent_cevent.AsyncResult.get File "src/gevent/event.py", line 327, in gevent._gevent_cevent.AsyncResult._raise_exception File "/usr/local/lib/python3.10/site-packages/gevent/_compat.py", line 66, in reraise raise value.with_traceback(tb) File "/usr/local/lib/python3.10/site-packages/gevent/threadpool.py", line 173, in __run_task thread_result.set(func(args, **kwargs)) socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 714, in urlopen httplib_response = self._make_request( File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 403, in _make_request self._validate_conn(conn) File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1053, in _validate_conn conn.connect() File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 363, in connect self.sock = conn = self._new_conn() File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f2d03e17010>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 798, in urlopen retries = retries.increment( File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='openaipublic.blob.core.windows.net', port=443): Max retries exceeded with url: /encodings/cl100k_base.tiktoken (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f2d03e17010>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/app/api/services/completion_service.py", line 205, in generate_worker Completion.generate( File "/app/api/core/completion.py", line 65, in generate rest_tokens_for_context_and_memory = cls.get_validate_rest_tokens( File "/app/api/core/completion.py", line 320, in get_validate_rest_tokens prompt_tokens = model_instance.get_num_tokens(prompt_messages) File "/app/api/core/model_providers/models/llm/openai_model.py", line 118, in get_num_tokens return max(self._client.get_num_tokens_from_messages(prompts) - len(prompts), 0) File "/usr/local/lib/python3.10/site-packages/langchain/chat_models/openai.py", line 552, in get_num_tokens_from_messages model, encoding = self._get_encoding_model() File "/usr/local/lib/python3.10/site-packages/langchain/chat_models/openai.py", line 530, in _get_encodingmodel encoding = tiktoken.encoding_for_model(model) File "/usr/local/lib/python3.10/site-packages/tiktoken/model.py", line 67, in encoding_for_model return get_encoding(model_encoding_name) File "/usr/local/lib/python3.10/site-packages/tiktoken/registry.py", line 63, in get_encoding enc = Encoding(constructor()) File "/usr/local/lib/python3.10/site-packages/tiktoken_ext/openai_public.py", line 64, in cl100k_base mergeable_ranks = load_tiktoken_bpe( File "/usr/local/lib/python3.10/site-packages/tiktoken/load.py", line 114, in load_tiktoken_bpe contents = read_file_cached(tiktoken_bpe_file) File "/usr/local/lib/python3.10/site-packages/tiktoken/load.py", line 46, in read_file_cached contents = read_file(blobpath) File "/usr/local/lib/python3.10/site-packages/tiktoken/load.py", line 24, in read_file return requests.get(blobpath).content File "/usr/local/lib/python3.10/site-packages/requests/api.py", line 73, in get return request("get", url, params=params, kwargs) File "/usr/local/lib/python3.10/site-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, kwargs) File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, send_kwargs) File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 519, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='openaipublic.blob.core.windows.net', port=443): Max retries exceeded with url: /encodings/cl100k_base.tiktoken (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f2d03e17010>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')) DEBUG:root:error: {'error': 'ConnectionError', 'description': "HTTPSConnectionPool(host='openaipublic.blob.core.windows.net', port=443): Max retries exceeded with url: /encodings/cl100k_base.tiktoken (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f2d03e17010>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))"} ERROR:root:internal server error. Traceback (most recent call last): File "/app/api/controllers/console/app/completion.py", line 164, in generate for chunk in response: File "/app/api/services/completion_service.py", line 415, in generate cls.handle_error(result) File "/app/api/services/completion_service.py", line 519, in handle_error raise Exception(description) Exception: HTTPSConnectionPool(host='openaipublic.blob.core.windows.net', port=443): Max retries exceeded with url: /encodings/cl100k_base.tiktoken (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f2d03e17010>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')) ERROR:app:Exception on /console/api/apps/933bf7bb-bb64-4f85-b3b0-30ec8303090c/chat-messages [POST] Traceback (most recent call last): File "/app/api/controllers/console/app/completion.py", line 164, in generate for chunk in response: File "/app/api/services/completion_service.py", line 415, in generate cls.handle_error(result) File "/app/api/services/completion_service.py", line 519, in handle_error raise Exception(description) Exception: HTTPSConnectionPool(host='openaipublic.blob.core.windows.net', port=443): Max retries exceeded with url: /encodings/cl100k_base.tiktoken (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f2d03e17010>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')) 有大佬知道为什么嘛

xusenlinzy commented 1 year ago

看样子是网络问题

panmyuan commented 1 year ago

看样子是网络问题

为啥会连openaipubilic.blob.core.windows.net呀？

xusenlinzy commented 1 year ago

应该是转向量的时候 cl100k_base.tiktoken 这个报错了，你是在无网络环境下使用的吗

panmyuan commented 1 year ago

应该是转向量的时候 cl100k_base.tiktoken 这个报错了，你是在无网络环境下使用的吗

是的，封闭的网络调用的，为啥还要调这个呢？，我的理解是不需要这些东西的，直接对话

xusenlinzy commented 1 year ago

因为 langchain 的 OpenAIEmbeddings 需要这个东西，dify 用的就是这个

panmyuan commented 1 year ago

因为 langchain 的 OpenAIEmbeddings 需要这个东西，dify 用的就是这个

好的，感谢解答哈

还有个疑惑哈，我选择chatglm作为模型，为什么还要去连接huggingface去调用tokenizer?

xusenlinzy commented 1 year ago

是不是你的模型提供商选的是chatglm？启动了模型后端的话，模型提供商选择OpenAI，别选其他的

MichaelPorter0 commented 1 year ago

因为 langchain 的 OpenAIEmbeddings 需要这个东西，dify 用的就是这个

选用dify 内网环境，有什么办法绕过这个吗

xusenlinzy / api-for-open-llm

怎么能对接dify，应该修改哪里呢？ #1