Potentially add workflow to have an automated testing suite?

[ArvinSKushwaha]

What would automated testing entail in this circumstance?

• A GitHub action that uses repository secrets (the OpenAI token) to run a series of tests on the project code.
• The tests could be comprised of a series of code snippets with different (relatively minor) flaws, with each paired with a set of expected outputs.
  • After passing the flawed code through the module code (with an expected output), some appropriate metric, like the Levenshtein distance (which may still be computationally feasible given the size of the programs?) can be used to compare the similarity between the flawed code and the output code as well as the similarity between the output code's output and the expected output. The relative distance of the two pairs of strings could potentially be used as a fitness score?
• It might even be possible to have GPT-4 score its own code, which would certainly be intriguing.

There are two major downsides to having an automated testing suite for a project like this:

• Tokens: an automated testing suite (especially on every push or pull_request) would cost a lot. Yikes!
• Safety: Executing code as a response to GPT-4 sounds like a big security risk. Also yikes!

However, it is still possible to build a workflow to perform automated testing, as GitHub actions has the workflow_dispatch option for manually running the action (which then automagically performs testing). Also, if concerned about security, it's not impractical to spin up a docker container for each test (though I reckon the github action runners do this anyways, so the effort may be redundant).

Any opinions?

[ArvinSKushwaha]

If interested, I may try to follow up on this issue with a PR to create a workflow that performs this task.

[siddhantdubey]

Yeah, testing would be really expensive. Really, really, expensive. I think just a set of more thorough tests that one could run locally if they wish would be nice.

[ArvinSKushwaha]

That's fair, I'll let you know if I have a prototype.