Add notes to sidekick about how to use it to write unit tests

[hanselke]

have you thought about how to hook the sidekick 'add feature/fix bug' to unit tests?

I'm thinking that just plain saving the file is bad, given the LLM isnt perfect. However if you reduce the scope of the problem to the human having to ensure the tests are correct first, we could just run loops on the new code updates, finally saving only when tests pass. Could even add instrumentation to it to test different implementations to get the 'best'.

[TechNickAI]

Some clarity on your question

Are you saying to build in a loop mechanism for sidekick of build feature, write tests, iterate until tests pass?

[hanselke]

yes. But we iterate over the 'feature' not the tests, until tests pass.

Think iterating on tests until they pass given current working code...is... something that i'd avoid building into the tooling. Heh users are going to use it thinking its magic and end up having tests that bake in their bugs.

Ideally it would actually write the tests, have the human verify it.

but we could start off with tests as a separate thing, making the 'build feature' dependent on tests being there to begin with. Call it a build to test feature.

I'm coming from a TDD approach where we write tests first.