[Proposal] Train a custom model

[winston-yallow]

Problem

ChatGPT was trained on data up until 2021, which is before Godot 4 and GDScript 2.0 were released. So it has no idea how they work. It mostly makes up stuff that looks similar to something that could work, but doesn't. It is an incredible frustrating experience to use this tool on a larger project.

It also does not transfer different concepts. It tries to translate scripts 1:1. It neglects, that in different engines you need to take different approaches overall.

Solution

A way to mitigate this would be to train a ML model directly on migrated projects. ChatGPT is specifically trained for general purpose human interactions. However, there are better ways (see GitHub Copilot) to train models directly for interacting with source code. A custom offline Model would also fix the potential data leaks mentioned #20

[KnedlikMCPE]

Instead of training a custom model, which is very expensive both in terms of dev time and computation, why not try using open-interpreter?

That actually leads me to why I visited this repo - I'd like to volunteer to trying out an implementation using Open Interpreter instead of ChatGPT, hoping that it could run the scripts (in Godot's case) or check for compile errors (in Unreal's case).

[winston-yallow]

From a quick glance at open-interpreter, here are a few thoughts:

• by default this seems to run code locally without any form of sandboxing
• while it can generate code from language, it would need to be tested how well it can generate code from code
• it would be best to find a way to run it sandboxed, maybe via a docker container?
• generally seems more focused on running code than generating clean code, so not sure about code quality
• it would be neat if the underlying LLM could be used a bit more directly, to not go through their tooling that is more targeted at usage by humans to execute code locally

[stuaxo]

You may want to collect examples of things that work and those that don't and the corrections needed.

Since this can't be exactly the code, people would need a way to submit examples to some open repo, that show the problems and solutions, then use them in training maybe.

[MarkusJLechner]

I came across some information that might be relevant. Has there been any consideration for fine-tuning GPT? OpenAI's blog suggests that fine-tuning can enhance performance. Training on the Godot 4 docs might be beneficial.

Also, I found that the OpenAI API doesn't store data for training, which might address issue #20. More details are in this OpenAI article.