search

Imperial-EE-Microsoft / co_op_translator

Easily automate multilingual translations for your projects with co_op_translator, powered by advanced LLM technology.
https://techcommunity.microsoft.com/t5/educator-developer-blog/localizing-github-repositories-with-llms/ba-p/4216434
MIT License
1 stars 2 forks source link

Implement synchronous markdown translation with try-catch mechanism to ensure complete processing of each chunk #35

Closed skytin1004 closed 1 month ago

skytin1004 commented 1 month ago

Description

Currently, this project uses an asynchronous approach for both image and markdown translations. This process breaks down long documents into smaller chunks based on token count for translation and then merges them back together. However, I've noticed an issue where some chunks are lost during translation, likely due to timeout errors. This results in incomplete translations and inconsistencies in the final output.

Cause

The issue seems to be related not to the length of the document but to the fact that some chunks are not properly received due to timeout errors during the asynchronous processing. When a chunk is not fully translated within a certain time frame, it can cause the subsequent chunks to be processed incorrectly or skipped altogether.

Solution

To address this, I plan to switch markdown translation to synchronous processing and implement a try-catch mechanism. This approach will ensure that each chunk is fully translated and received before moving on to the next.

This issue was discussed with @leestott and suggested as an improvement to move to synchronous processing with a try catch so if chunk x does complete don't move to chunk y and provide a error

leestott commented 1 month ago

Closing issue as we have migrated to a Microsoft Repo