search

machinewrapped / gpt-subtrans

Open Source project using LLMs to translate SRT subtitles
Other
347 stars 40 forks source link

Upper limiting the previous context #124

Closed yuxi-liu-wired closed 7 months ago

yuxi-liu-wired commented 7 months ago

I am translating some documentaries where there are no distinct scenes, and I noticed that it is costing more than I expected.

I guess that the translator sends all summaries of all batches in the current scene. It can be pretty expensive, because the prompt grows linearly, and so the total cost grows quadratically with the length of the documentary.

My proposed solution is to only send a limited number of previous batches. This keeps the cost of a long scene down.

machinewrapped commented 7 months ago

There should be an upper limit already, max_context_summaries in Options.py - it defaults to 10 and is applied in GetBatchContext in SubtitleFile.py... try a lower number if the summaries are getting long (GPT4 tends to write much more detailed summaries than GPT3.5).

There's no command line option to set it at the moment (easy to add one), but it can be set in the GUI (advanced settings) or as an environment variable in a .env file.

yuxi-liu-wired commented 7 months ago

Thanks for the reply. Perhaps this can be added to the terminal option.

machinewrapped commented 7 months ago

Sure, easily done! https://github.com/machinewrapped/gpt-subtrans/commit/30fa3a76c9c40ff557087a073ec87fd947ed70e8