When llm system prompt is relatively long, it seems that the workflow cannot be saved

[Akifyss]

Self Checks

• [X] This is only for bug report, if you would like to ask a quesion, please head to Discussions.
• [X] I have searched for existing issues search for existing issues, including closed ones.
• [X] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [X] Pleas do not modify this template :) and fill in all the required fields.

Dify version

0.6.0-fix1

Cloud or Self Hosted

Cloud

Steps to reproduce

Add an llm node in the workflow. When the llm system prompt is relatively long, save fails, and upon reloading the page, it is found that modifications are lost.

✔️ Expected Behavior

No response

❌ Actual Behavior

No response

[dosubot[bot]]

Hey there, @Akifyss! 🚀 Great to see you diving into the intricacies of Dify again. How's everything going on your end?

Thank you for providing detailed information about the issue you're experiencing. Based on the context provided, it seems that the issue arises due to a character limit set for the LLM system prompt in the Dify codebase. This limit is determined by the model_context_tokens minus the sum of max_tokens and prompt_tokens. If the remaining tokens (rest_tokens) are less than 0, an InvokeBadRequestError is raised, indicating that the query or prefix prompt is too long. You can find this mechanism in the base_app_runner.py file.

When the LLM system prompt exceeds the character limit, an error message is displayed in the UI. This is handled by the InvokeBadRequestError in the _invoke_error_mapping property and its usage in the _generate method. If the response status code is not 200, an InvokeBadRequestError is raised with the message including the status code and the response text, which likely contains details about the character limit being exceeded. This is implemented in the llm.py file.

However, it's unclear how the Dify UI handles the InvokeBadRequestError and displays it to the user. This information might be available elsewhere or I might have missed it in the repository.

The Dify UI handles long system prompts in the LLM node by displaying the current character count of the input. This is evident from the code snippet provided, particularly in the footer section of the BlockInput component. The character count is shown in a div element at the bottom of the input area, updating dynamically as the user types. This feature serves as an indirect warning mechanism about the character limit by keeping the user informed about the length of their input in real-time. However, there is no explicit warning or error mechanism in the provided code that prevents saving the workflow or alerts the user when the character limit is exceeded, other than the dynamic character count display. You can find this in the block-input/index.tsx file.

To increase the model_context_tokens limit in the Dify codebase for accommodating longer LLM system prompts, you would need to adjust the max_tokens parameter within the YAML configuration file for the specific model you are working with. In the provided context of the spark-2.yaml file, the max_tokens parameter is already defined with a default value of 2048 and a maximum allowable value of 8192. To increase the limit, you can either change the default value or instruct users to specify a higher value up to the maximum of 8192 when they are configuring or invoking the model, assuming the change is intended to allow for longer inputs or outputs. If you need the limit to be higher than 8192, you would have to modify the max value to your desired higher limit, provided the underlying system and model support it. You can find this in the spark-2.yaml file.

I hope this information helps you understand the issue better. If you have any further questions or need more clarification, please feel free to ask.

Sources

• api/core/app/apps/base_app_runner.py
• api/core/model_runtime/model_providers/triton_inference_server/llm/llm.py
• web/app/components/base/block-input/index.tsx
• api/core/model_runtime/model_providers/spark/llm/spark-2.yaml

#### About Dosu

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.

[takatost]

Can you please provide the response of the request to /console/api/apps/:app_id/workflows/draft at that time?

[Akifyss]

Can you please provide the response of the request to /console/api/apps/:app_id/workflows/draft at that time?

Thank you for your reply But sorry, I am not a professional and do not quite understand how to get the above information? Could you please provide some detailed steps?

[takatost]

Can you please provide the response of the request to /console/api/apps/:app_id/workflows/draft at that time?

Thank you for your reply But sorry, I am not a professional and do not quite understand how to get the above information? Could you please provide some detailed steps?

@dosubot pls provide the steps to get the response of the above interface request in Chrome.

[dosubot[bot]]

To retrieve the response of the /console/api/apps/:app_id/workflows/draft request in Chrome, especially when encountering an issue with saving long LLM system prompts in Dify version 0.6.0-fix1, you can use the Chrome Developer Tools. Here's how you can do it:

1. Open Chrome and navigate to the page where you're trying to save the long LLM system prompts.
2. Right-click anywhere on the page and select "Inspect" to open the Chrome Developer Tools.
3. Go to the "Network" tab in the Developer Tools window.
4. Make sure the record button (the red circle at the top left of the Network tab) is on. If it's grey, click it to start recording network activity.
5. Perform the action that triggers the /console/api/apps/:app_id/workflows/draft request. This could be saving your workflow, for example.
6. Look for the request named /console/api/apps/:app_id/workflows/draft in the list of network requests. You might need to filter by "XHR" requests to find it more easily.
7. Click on the request to view its details. You can find the response in the "Response" tab.

This method allows you to inspect the HTTP response directly from the server, which can be very helpful for debugging issues related to network requests in web applications.

As for the expected format and data type of the LLM system prompt in the Dify codebase, it is a string that contains instructions, examples, and specific formatting requirements for the output. The prompt should guide the user or the system in generating outputs that are structured in a particular way, often requiring the output to be in JSON format. The data within the prompts is textual, combining instructions, examples, and sometimes variables within a structured narrative to guide the generation of outputs that meet specific criteria or formats.

You can find more details in the Dify codebase, specifically in the api/controllers/console/app/workflow.py and api/core/prompt/prompts.py files.

Sources

• api/controllers/console/app/workflow.py
• api/core/prompt/prompts.py

#### About Dosu

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.

[Akifyss]

The following information was returned

<!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>
<title>Attention Required! | Cloudflare</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1" />
<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/cf.errors.css" />
<!--[if lt IE 9]><link rel="stylesheet" id='cf_styles-ie-css' href="/cdn-cgi/styles/cf.errors.ie.css" /><![endif]-->
<style>body{margin:0;padding:0}</style>

<!--[if gte IE 10]><!-->
<script type="7df4c55e98b1e061f7343fd6-text/javascript">
  if (!navigator.cookieEnabled) {
    window.addEventListener('DOMContentLoaded', function () {
      var cookieEl = document.getElementById('cookie-alert');
      cookieEl.style.display = 'block';
    })
  }
</script>
<!--<![endif]-->

</head>
<body>
  <div id="cf-wrapper">
    <div class="cf-alert cf-alert-error cf-cookie-error" id="cookie-alert" data-translate="enable_cookies">Please enable cookies.</div>
    <div id="cf-error-details" class="cf-error-details-wrapper">
      <div class="cf-wrapper cf-header cf-error-overview">
        <h1 data-translate="block_headline">Sorry, you have been blocked</h1>
        <h2 class="cf-subheadline"><span data-translate="unable_to_access">You are unable to access</span> dify.ai</h2>
      </div><!-- /.header -->

      <div class="cf-section cf-highlight">
        <div class="cf-wrapper">
          <div class="cf-screenshot-container cf-screenshot-full">

              <span class="cf-no-screenshot error"></span>

          </div>
        </div>
      </div><!-- /.captcha-container -->

      <div class="cf-section cf-wrapper">
        <div class="cf-columns two">
          <div class="cf-column">
            <h2 data-translate="blocked_why_headline">Why have I been blocked?</h2>

            <p data-translate="blocked_why_detail">This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.</p>
          </div>

          <div class="cf-column">
            <h2 data-translate="blocked_resolve_headline">What can I do to resolve this?</h2>

            <p data-translate="blocked_resolve_detail">You can email the site owner to let them know you were blocked. Please include what you were doing when this page came up and the Cloudflare Ray ID found at the bottom of this page.</p>
          </div>
        </div>
      </div><!-- /.section -->

      <div class="cf-error-footer cf-wrapper w-240 lg:w-full py-10 sm:py-4 sm:px-8 mx-auto text-center sm:text-left border-solid border-0 border-t border-gray-300">
  <p class="text-13">
    <span class="cf-footer-item sm:block sm:mb-1">Cloudflare Ray ID: <strong class="font-semibold">871a0bfe3d3e17ec</strong></span>
    <span class="cf-footer-separator sm:hidden">&bull;</span>
    <span id="cf-footer-item-ip" class="cf-footer-item hidden sm:block sm:mb-1">
      Your IP:
      <button type="button" id="cf-footer-ip-reveal" class="cf-footer-ip-reveal-btn">Click to reveal</button>
      <span class="hidden" id="cf-footer-ip">2602:feda:f1d6:12ac:44dc:bb99:752c:ff09</span>
      <span class="cf-footer-separator sm:hidden">&bull;</span>
    </span>
    <span class="cf-footer-item sm:block sm:mb-1"><span>Performance &amp; security by</span> <a rel="noopener noreferrer" href="https://www.cloudflare.com/5xx-error-landing" id="brand_link" target="_blank">Cloudflare</a></span>

  </p>
  <script type="7df4c55e98b1e061f7343fd6-text/javascript">(function(){function d(){var b=a.getElementById("cf-footer-item-ip"),c=a.getElementById("cf-footer-ip-reveal");b&&"classList"in b&&(b.classList.remove("hidden"),c.addEventListener("click",function(){c.classList.add("hidden");a.getElementById("cf-footer-ip").classList.remove("hidden")}))}var a=document;document.addEventListener&&a.addEventListener("DOMContentLoaded",d)})();</script>
</div><!-- /.error-footer -->

    </div><!-- /#cf-error-details -->
  </div><!-- /#cf-wrapper -->

  <script type="7df4c55e98b1e061f7343fd6-text/javascript">
  window._cf_translation = {};

</script>

<script src="/cdn-cgi/scripts/7d0fa10a/cloudflare-static/rocket-loader.min.js" data-cf-settings="7df4c55e98b1e061f7343fd6-|49" defer></script></body>
</html>

[Akifyss]

Longer texts will trigger an error, but if I delete parts of the original text, I can save it successfully and get the following information:{"result": "success", "updated_at": 1712661419}

[Akifyss]

I've found the real triggering reason; it should have nothing to do with the length of the prompt. As long as there is a " Githubissues.

Githubissues is a development platform for aggregating issues.