timeout when add llama3.1 model as text embedding model

[caolizhong-pm]

Self Checks

• [X] This is only for bug report, if you would like to ask a question, please head to Discussions.
• [X] I have searched for existing issues search for existing issues, including closed ones.
• [X] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• [X] Please do not modify this template :) and fill in all the required fields.

Dify version

0.8.2

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

timeout when add llama3.1 70b model as text embedding model, but local llama3 8b model has no issue

✔️ Expected Behavior

llama3.1 70b model can be added as text embedding model

❌ Actual Behavior

saving failed in GUI and timeout in backend log

[dosubot[bot]]

Hello @caolizhong-pm! I'm here to help you with any bugs, questions, or contributions you have for the project. Let's work together to solve this issue!

I found a similar issue regarding timeouts when adding larger models. The timeout issue when adding the llama3.1 70b model as a text embedding model, while the local llama3 8b model works without issues in Dify version 0.8.2, is likely due to the larger size and complexity of the llama3.1 70b model. Larger models require significantly more computational resources and time to process requests, which can lead to timeouts if the system is not adequately provisioned to handle such demands ^[1].

To address this, you can try the following steps:

1. Increase Timeout Settings: Adjust the default timeout settings in your configuration file. The default settings are:

   • API_TOOL_DEFAULT_CONNECT_TIMEOUT=10 (10 seconds for connection timeout)
   • API_TOOL_DEFAULT_READ_TIMEOUT=60 (60 seconds for read timeout)

   You can increase these values to allow more time for the larger model to process requests ^[2].

2. Resource Allocation: Ensure that your system has sufficient computational resources (CPU, GPU, memory) to handle the larger model. You may need to allocate more resources or optimize your current setup.

3. Model Optimization: Consider using a more optimized version of the model if available, or explore other models that provide similar capabilities but with lower resource requirements.

If the issue persists, you may need to review the detailed steps for troubleshooting model configuration issues in Dify.

To continue talking to Dosu, mention @dosu.

[crazywoola]

Are there any logs or errors that can help us to investigate?

[caolizhong-pm]

docker-api-1         | Traceback (most recent call last):
docker-api-1         |   File "/app/api/controllers/console/workspace/models.py", line 151, in post
docker-api-1         |     model_provider_service.save_model_credentials(
docker-api-1         |   File "/app/api/services/model_provider_service.py", line 250, in save_model_credentials
docker-api-1         |     provider_configuration.add_or_update_custom_model_credentials(
docker-api-1         |   File "/app/api/core/entities/provider_configuration.py", line 390, in add_or_update_custom_model_credentials
docker-api-1         |     provider_model_record, credentials = self.custom_model_credentials_validate(model_type, model, credentials)
docker-api-1         |   File "/app/api/core/entities/provider_configuration.py", line 370, in custom_model_credentials_validate
docker-api-1         |     credentials = model_provider_factory.model_credentials_validate(
docker-api-1         |   File "/app/api/core/model_runtime/model_providers/model_provider_factory.py", line 123, in model_credentials_validate
docker-api-1         |     model_instance.validate_credentials(model, filtered_credentials)
docker-api-1         |   File "/app/api/core/model_runtime/model_providers/ollama/text_embedding/text_embedding.py", line 123, in validate_credentials
docker-api-1         |     raise CredentialsValidateFailedError(f"An error occurred during credentials validation: {str(ex)}")
docker-api-1         | core.model_runtime.errors.validate.CredentialsValidateFailedError: An error occurred during credentials validation: HTTPConnectionPool(host='xxxxxxxxx.net', port=8080): Read timed out. (read timeout=300)

[crazywoola]

It seems to be a general network issue to me. Can you call this model with api?

[caolizhong-pm]

yes, call model is normal and also able to add this model as chat model

[caolizhong-pm]

Hi, any update

[crazywoola]

Nope