OpenBMB / RepoAgent

An LLM-powered repository agent designed to assist developers and teams in generating documentation and understanding repositories quickly.
Apache License 2.0
374 stars 64 forks source link

Handling `UnicodeDecodeError` During File Read Operation #19

Closed Umpire2018 closed 10 months ago

Umpire2018 commented 10 months ago

Description:

Encountered a UnicodeDecodeError while attempting to read content from a file that contains a mix of English and Chinese characters. The content was initially saved with utf-8 encoding but resulted in encoding errors when read back from the file.

Error Message:

  File "AI_doc\ai_doc\runner.py", line 341, in <module>
    runner.run()
  File "AI_doc\ai_doc\runner.py", line 165, in run
    self.process_file_changes(repo_path, file_path, is_new_file)
  File "AI_doc\ai_doc\runner.py", line 225, in process_file_changes
    markdown = file_handler.convert_to_markdown_file(file_path=file_handler.file_path)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    json_data = json.load(f)
                ^^^^^^^^^^^^
  File "Python\Python311\Lib\json\__init__.py", line 293, in load
    return loads(fp.read(),
                 ^^^^^^^^^
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xba in position 207: invalid start byte

Issue Details: The error occurred in the convert_to_markdown_file method, which suggests that the file content may have been incorrectly encoded or that the file contains a mix of encodings that are not properly handled by the standard utf-8 decoder.

Content Example:

        "synthesize_voice": {
            "type": "FunctionDef",
            "name": "synthesize_voice",
            "md_content": "**synthesize_voice����**���ú����Ĺ����ǽ�ָ���������ϳ�Ϊ�����������ϳɵ��������浽ָ�����ļ����С�\n\n�ú�������ϸ�������������£�\n\n- ���ȣ���voice_name_details��������ȡ�������ƺ��Ա���Ϣ���������ƺ��Ա���Ϣ֮��ʹ����������\"��\"��\"��\"���зָ���ͨ��rsplit�������������ƺ��Ա���Ϣ���룬��ʹ��rstrip����ȥ���Ա���Ϣĩβ���������š�Ȼ��ʹ��replace�������Ա���Ϣ�е�\"Ů��\"�滻Ϊ\"Ů\"����\"��ͯ\"�滻Ϊ\"ͯ\"���Լ��Ա�ı�ʾ��ʽ��\n\n- ���������������������õ�speech_config�����speech_synthesis_voice_name�����У��Ա��������ϳ�ʱʹ��ָ����������\n\n- Ȼ��ʹ��ѭ������������Դ����ij��ԡ�\n\n- ��ÿ�γ����У����ȳ�ʼ��SpeechSynthesizer���󣬲�����speech_config������\n\n- Ȼ��ʹ��os.path.join����������ļ��к��������ơ��Ա�ƴ�ӳ������Ƶ�ļ���·����\n\n- ���ţ�����AudioConfig���󣬽��ļ�·������filename������\n\n- Ȼ��ʹ��ָ����audio_config������ʼ��SpeechSynthesizer����\n\n- ����SpeechSynthesizer�����speak_text_async��������Ҫ�ϳɵ��ı���Ϊ�������룬��ʹ��get������ȡ�ϳɽ����\n\n- ���ϳɽ����reason���ԣ�����ϳɳɹ������ӡ�ϳɳɹ�����ʾ��Ϣ�������ء�\n\n- ����ϳɱ�ȡ�������ӡȡ����ԭ�򣬲�����ȡ����ԭ�������Ӧ�Ĵ�����\n\n- ��������쳣�����ӡ�쳣��Ϣ������ָ���������ӳ�ʱ���������ԡ�\n\n- ����ﵽ������Դ�����Ȼ�޷��ϳ����������ӡ�ϳ�ʧ�ܵ���ʾ��Ϣ��\n\n**ע��**��ʹ�øú���ʱ��Ҫע�����¼��㣺\n- ��Ҫ�ṩ�ϳ��������������ƺ��Ա���Ϣ��\n- ��Ҫ�ṩSpeechConfig������Ϊ���������ڸö��������ú��ʵ�������Ϣ��\n- ��Ҫ�ṩ����ļ��е�·����\n- ��Ҫָ��������Դ����������ӳ�ʱ�䡣\n\n**���ʾ��**������ɹ��ϳ��������������浽��ָ�����ļ����С�",
            "code_start_line": 42,
            "code_end_line": 85,
            "parent": null,
            "have_return": true,
            "code_content": "def synthesize_voice(voice_name_details, speech_config, output_folder, max_retries, retry_delay):\n    # Extract voice name and gender from the details\n    voice_name, gender = voice_name_details.rsplit('��', 1)\n    gender = gender.rstrip('��')\n    gender = gender.replace('Ů��', 'Ů').replace('��ͯ', 'ͯ')  # Simplify gender notation\n\n    # Set the voice name in the speech config.\n    speech_config.speech_synthesis_voice_name = f\"zh-CN-{voice_name}\"\n\n    for attempt in range(max_retries):\n        try:\n            # Initialize speech synthesizer.\n            synthesizer = SpeechSynthesizer(speech_config=speech_config)\n\n            # Get the path to the output audio file.\n            file_path = os.path.join(output_folder, f\"{voice_name}_{gender}.wav\")\n\n            audio_config = AudioConfig(filename=file_path)\n\n            # Use the synthesizer with the specified audio configuration\n            synthesizer = SpeechSynthesizer(speech_config=speech_config, audio_config=audio_config)\n\n            # Synthesize the voice name to a file.\n            result = synthesizer.speak_text_async(example_text).get()\n\n            # Check the result and break the loop if successful.\n            if result.reason == ResultReason.SynthesizingAudioCompleted:\n                print(f\"Speech synthesized for voice {voice_name} and saved to {file_path}\")\n                return\n            elif result.reason == ResultReason.Canceled:\n                cancellation_details = result.cancellation_details\n                print(f\"Speech synthesis canceled: {cancellation_details.reason}\")\n                if cancellation_details.reason == CancellationReason.Error:\n                    if cancellation_details.error_details:\n                        print(f\"Error details: {cancellation_details.error_details}\")\n                        raise Exception(cancellation_details.error_details)\n        except Exception as e:\n            print(f\"An error occurred: {e}. Retrying in {retry_delay} seconds.\")\n            time.sleep(retry_delay)\n        \n\n    print(f\"Failed to synthesize voice {voice_name} after {max_retries} attempts.\")\n",
            "name_column": 4
        }

A snippet from the file content includes function definitions and comments in both English and Chinese. The original content has been corrupted with a series of ���� characters, which are indicative of encoding issues.

Solution Discussed: To address this issue, Maybe using charset_normalizer to read the file in subsequent logic operations. This approach involves using charset_normalizer to re-read the file content successfully, detecting the correct encoding, and decoding the file content properly.

Proposed Changes to Workflow:

Additional Context: This solution aims to normalize the file content during the read operation without changing the initial file-saving behavior. By processing the encoding on read, we can handle files from various sources and encoding states more robustly.

Umpire2018 commented 10 months ago

Fix


from charset_normalizer import from_path

# 文件路径
file_path = '.project_hierarchy.json'

# 使用 charset_normalizer 读取和修复文件编码问题
matches = from_path(file_path)

# 获取最佳匹配结果
best_match = matches.best()

print(str(best_match))

Result

image