langgenius / dify

Dify is an open-source LLM app development platform. Dify's intuitive interface combines AI workflow, RAG pipeline, agent capabilities, model management, observability features and more, letting you quickly go from prototype to production.
https://dify.ai
Other
52.36k stars 7.64k forks source link

Add maxLengthTip to web and increase max-chunk-size #10981

Open xu-song opened 8 hours ago

xu-song commented 8 hours ago

Summary

  1. Add maxLengthTip: Maximum chunk length is ambiguous. It could be confused with string length.
  2. Increase max-chunk-size: With default gpt2-tokenizer, 1000 tokens is roughly equivalent to 400 CJK characters. It is not enough in most cases.

Screenshots

Screenshot 2024-11-22 at 17 04 06

Checklist

[!IMPORTANT]
Please review the checklist below before submitting your pull request.