thygrrr
commented
3 weeks ago Actual Conversation for Prompt 1 With Pieces (GPT-4 - Gemini Pro didn't seem to accept the context)
This is a complete fail - this is hallucinated code for some hypothetical, generic ECS, which would operate quite differently from how the library in the context operates.
Out of fairness, I also submitted the context in the exact same form as I submitted it to Claude, but the result is practically identical and completely off-target.
Actual Conversation for Prompt 1 with Claude
This is pretty much full marks again - this example would work as written; if the one occurrence of .Compile() were replaced with .Stream(). The foreach Method isn't the best practice method, but the Repo itself actually doesn't contain an example how to use the relatively unique Stream Views efficiently. (I didn't add the hand-written documentation to the context - I did that in other experiments and Claude was basically able to write short tutorial pages for the library almost independently.)
Software
JetBrains
Operating System / Platform
Windows
Your Pieces OS Version
9.0.5
Early Access Program
Kindly describe the bug and include as much detail as possible on what you were doing so we can reproduce the bug.
I have a library of around 7500 LoC (much of it comments/documentation) of modern C# that I set as context. The project is 100% C#.
Context for Reproduction
The sole context is in this specific sub-folder in my MIT-licensed repo: https://github.com/outfox/fennecs/tree/main/fennecs For the examples in this report, I added it as a Folder context most of the time
At the end I submitted it as a File context, with the entire C# contents of the directory concatenated into a markdown file, a small paragraph of opening instructions for the LLM, and with H2 for the file names. (this is my competing workflow with Claude)
It roughly looks like this. :)
"Bug" / Unexpected Behaviour
The prompts and conversations are of extremely low quality.
I tried various models and keep getting very tangentially related, sometimes completely unrelated (different programming language, etc.), often gibberish-like suggestions. (interacting with the model is also bordering on the impossible)
Contrasting Expectation / Competition
To put this into perspective, this is the 1-shot I get with Claude 3 in my own workflow (which submits a simple markdown summary / concatenation of the code). This earns practically full marks in focus, correctness, and "understanding" of what a new user would like to do first with this library.
This, or roughly this, Quality of initial prompts would show me that Pieces has built a reasonable rapport and good bird's-eye view of the submitted context.