The code execution module in the workflow reports an error:Failed to parse result

Sakura4036 commented 3 months ago

Self Checks

[X] This is only for bug report, if you would like to ask a question, please head to Discussions.
[X] I have searched for existing issues search for existing issues, including closed ones.
[X] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
[X] Please do not modify this template :) and fill in all the required fields.

Dify version

0.6.11

Cloud or Self Hosted

Self Hosted (Source)

Steps to reproduce

I called the literature search API to get the titles and abstracts of the relevant literature, and the resulting json string needs to be parsed by the code execution module.

I tried to modify the CODE EXECUTION CONFIGURATION in the env (as shown below), however, the code execution module is still unable to process the large amount of data.

✔️ Expected Behavior

The code execution module successfully parses the string obtained from the API.

I call the literature search API to get the titles and abstracts of the relevant literature, if I only get the abstract information of 20 literature, the code execution module parses and executes successfully.

❌ Actual Behavior

however, due to the fact that the number of literature searched is too large (1,000 articles, about 2 million characters), the parsed data can not be parsed and output.

dosubot[bot] commented 3 months ago

To resolve the issue where the code execution module fails to parse large JSON strings, you need to ensure that the JSON string length does not exceed the MAX_STRING_LENGTH limit. The MAX_STRING_LENGTH is set to 80,000 characters by default, but you can adjust this limit by setting the CODE_MAX_STRING_LENGTH environment variable.

Here's how you can set the environment variable to increase the maximum allowed string length:

import os

# Set the environment variable to increase the maximum string length
os.environ['CODE_MAX_STRING_LENGTH'] = '160000'  # Example: setting it to 160,000 characters

Make sure to set this environment variable before running your code execution module. This will allow the module to handle larger JSON strings.

Additionally, ensure that the _check_string method in your CodeNode class is properly handling the string length validation:

def _check_string(self, value: str, variable: str) -> str:
    """
    Check string
    :param value: value
    :param variable: variable
    :return:
    """
    if not isinstance(value, str):
        raise ValueError(f"Output variable `{variable}` must be a string")

    if len(value) > MAX_STRING_LENGTH:
        raise ValueError(f'The length of output variable `{variable}` must be less than {MAX_STRING_LENGTH} characters')

    return value.replace('\x00', '')

By increasing the MAX_STRING_LENGTH limit and ensuring the validation logic is in place, you should be able to parse larger JSON strings successfully ^[1].

To continue talking to Dosu, mention @dosu.