Other models to fine-tune

[viktor-ferenczi]

It is not an issue, more of a bunch of suggestions / observations.

While experimenting with openly available LLMs I found the following models useful for coding:

• WizardCoder (a StarCoder fine-tune itself)
• Airoboros (tested the Llama 1 based 65B one, now there is one based on Llama 2)

They were quite competent in coding (compared to other openly available LLMs) even when loaded as 8 bit GGML.

I think fine-tuning the new Airoboros-l2 70B one may be worth a try. However, its license is ambiguous, so careful.

Alternatively fine-tuning WizardCoder for Godot may end up as a smaller (more usable) model at surprisingly good quality.

The author of Airoboros has some very good insights on the HF page of his Airoboros-65B model. It is also worth reading the paper behind the WizardCoder model, they came up with completely synthetic training data without having to use another LLM to produce them, which is good due to licensing limitations.

[viktor-ferenczi]

Further observations:

• LLMs are slightly better to produce strictly typed code than dynamically typed. Asking for the inclusion of type-hints seems to help, at least in case of Python.
• Smaller LLMs require a lot more tinkering with the prompt to get usable (or easily fixable) code out of them.
• Fine-tuning for each programming language (or family of languages) separately seems to make perfect sense if we want smaller models to be usable, cheaper to train and use for inference.
• We need to train the model with shader code as well. Due to its similarity with GLSL it should be perfectly possible.

A good prompt for a non-trivial coding test is the following:

You are an expert Python developer. 
Your task is to write a Python 3 function to identify duplicate files in a folder and return a summary of them.

Requirements:
- At any depth in the subdirectory structure.
- Two files are duplicates if they have the same size and contents.
- Optimization: Files with a unique size can be skipped, because they must be unique.
- File contents can be checked based on their SHA256 hashes (checksums).
- Do not read whole files into memory, calculate the hash in 32kB chunks.
- The risk of a hash collision is acceptable in this use case.
- Must find all duplicate files.
- Must NOT delete any files.
- The return value of the function must be a dictionary where keys are the file size and checksum in a tuple, values are the list of paths.
- The solution must work on both Windows and UNIX (Linux, MAC).

Further instructions:
- Add only very concise comments into the code wherever it is absolutely necessary.
- Keep the code in each function short and as simple as possible.
- Avoid deep nesting of flow control.
- Avoid assigning variables which are not used afterwards.
- Structure the code to be very easy to read and understand by humans.
- You are an expert developer, you can code this simple task very well.
- Add type hints to all function parameters, return values and variables.
- Provide only the code and nothing else.

What would be a good coding test for Godot 4's GDScript?

[minosvasilias]

Hey, thanks for the comprehensive thoughts and feedback, this is good stuff to read. Let me go through these one by one.

Regarding other models, i am definitely interested in Wizardcoder in particular. The reason that finetune hasn't happened yet is a mix of me not having had much time recently (due to a job change) and also the fact that i think the biggest gains in this project will at this point not come from better models, but better data. At least for now.

I still think the general idea of annotating human-created code for instruct datasets is a good one, but the current dataset has some very obvious drawbacks. The two biggest problems i believe are:

• Scope issues due to splitting each source script into individual functions, resulting in the model learning that it's okay to not define variables, use non-existent custom classes, or call do_x() instead of implementing the logic for x.
• Overrepresentation of plugin code. Since adding plugins to a Godot project is not a plain import, but copies the files into an addons folder (which is never gitignored), there's tons of repos repeating the same code over and over again. While exact duplicates are removed, that still leaves lots of minor changes to those methods being included regardless.

So these are issues i wanted to address before doing another training run. I've been working on this a bit and have some approaches i've been testing to fix the scope issue in particular. But yeah, wizardcoder would most likely be the next model to test. (until something more interesting comes out)

Agree with all your other points. Especially training on shader-code, that is something i'm very interested in as well. (and not just for Godot) Thinking this should probably be a separate model considering it's another language.

One problem with shader code is that i found it much harder to generate accurate instructions for it. Since all logic serves to create a visual (most of the time), and instructions in the real world for such a model would likely be things like "create a cel-shaded water shader", i think a different approach for instruction generation is needed here. Been experimenting with image-to-text models, which i think works well, though the best ones tend to not be open-source unfortunately, limiting the quality of descriptions a bit. But i've done some work on it already and will publish something if i can complete it and get some decent results.

And regarding the coding test: I didn't really want to rely on a single datapoint, which is why i went with the 50 different tests in my evaluation set. It's not perfect by any means, but already averages out a lot of bias you would otherwise find. As for the rules present in your prompt: The most common failure cases i found in GPT models that could be reduced with that kind of few-shot prompting were:

• Generation of Python import statements
• Confusion of GDScript 1 and 2 syntax
• Bad memorization of native constants (like Color.RED)
• Poor understanding of node types and appropriate usage (i.e. 2D and 3D specific methods etc.)