code type included in code

aristotle-tek commented 1 year ago

Simple issue - when it generates a html file, for example, rather than starting the file with <!DOCTYPE html>, it starts with:

\```html
<!DOCTYPE html>

Or similarly a python file it starts with the word "python"

hanchchch commented 1 year ago

Hmm that's interesting behavior. It might because our description prompt for CodeEditor.WRITE tool wasn't clear enough. Thank you very much for sharing this issue!

hanchchch commented 1 year ago

Can you share me the prompt that you used when this issue occurred? It'd help me to reproduce the issue.

aristotle-tek commented 1 year ago

It happened a few times. Here are two commands: curl -X POST -H "Content-Type: application/json" -d '{"key": "sessionid", "files": [], "query": "Please develop and serve a simple web CARBONFOOTPRINTCALCULATOR app. The user can list the number and distance of round-trip flights they have taken, and it will report the estimated kilograms of CO2 emitted. I want it to have neuromorphism-style. The ports you can use are 4000 and 7000. Cannot use PORT variable in environment."}' http://localhost:8000/command

curl -X POST -H "Content-Type: application/json" -d '{"key": "sessionid", "files": [], "query": "Please develop and serve a simple web TODO app. The user can list all TODO items and add, delete each TODO item. I want it to have neuromorphism-style. The ports you can use are 4000 and 7000. Cannot use PORT variable in environment."}' http://localhost:8000/command

The second one is basically just your prompt, so strange it would cause an issue. But see the response:

Input: python todo_app.py
eval  | Realtime Terminal Output: 
eval  | stderr   File "/app/playground/todo_app.py", line 28
eval  |     ```
eval  |     ^
eval  | SyntaxError: invalid syntax
eval  | Observation:   File "/app/playground/todo_app.py", line 28
eval  | ...     ```
eval  | ...     ^
eval  | ... SyntaxError: invalid syntax
eval  | ... 
eval  | Thinking...
eval  | Plan:  
eval  | - [ ] Remove the markdown code block syntax from the end of the Python file
eval  | - [ ] Run the TODO app on either port 4000 or 7000
eval  | - [ ] Test the app to ensure it works as expected
eval  | What I Did:  Removed the markdown code block syntax from the beginning of the Python file
eval  | Action: CodeEditor.PATCH
eval  | Input: todo_app.py|28,1|28,4|
eval  | Observation: successfully wrote 0, deleted 3
eval  | Thinking...
eval  | Plan:  
eval  | - [ ] Run the TODO app on either port 4000 or 7000
eval  | - [ ] Test the app to ensure it works as expected
eval  | What I Did:  Removed the markdown code block syntax from the end of the Python file
eval  | Action: Terminal
eval  | Input: python todo_app.py
eval  | Realtime Terminal Output: 
eval  | stderr Traceback (most recent call last):
eval  |   File "/app/playground/todo_app.py", line 1, in <module>
eval  |     n
eval  | NameError: name 'n' is not defined

etc

cvarrichio commented 1 year ago

I tried the same prompt and received the exact same error, although it appears that EVAL may have been able to fix it in further steps on its own.

hanchchch commented 1 year ago

Thanks for sharing! I'll take a look on it.

hanchchch commented 1 year ago

The second one is basically just your prompt, so strange it would cause an issue.

Yeah it's strange, we've tried that prompt for tens of times but never happened. I'll try with a higher temperature since LLM may throw more various outputs on same prompt.

hanchchch commented 1 year ago

@aristotle-tek @cvarrichio Hi, I fixed the tool description prompt a little bit. I tested with your prompt for many times (about 5 times) and worked perfectly. Can you check it out and test it? I'd appreciate it. #16

12charl commented 1 year ago

What's the best code editor for Android Used in writing website

12charl commented 1 year ago

What's the best code editor for Android Used in writing website

12charl commented 1 year ago

What's the best code editor for Android Used in writing website

corca-ai / EVAL

code type included in code #15