search

princeton-nlp / SWE-agent

SWE-agent takes a GitHub issue and tries to automatically fix it, using GPT-4, or your LM of choice. It solves 12.47% of bugs in the SWE-bench evaluation set and takes just 1 minute to run.
https://princeton-nlp.github.io/SWE-agent/
MIT License
13.29k stars 1.3k forks source link

rename 'history' field in trajectory file to 'demonstration'? #747

Closed ofirpress closed 2 weeks ago

ofirpress commented 2 weeks ago

we have a history field in the trajectory files.

this field does not contain the history, it just has the demonstration.

this might be confusing to some people. we should've called that field 'demonstration'.

i don't think we should go for the hassle of fixing this, maybe we should just close this issue.

i do want this issue to exist so that people can see this if they search for information on google.

if you need an explainer on what the demonstration is, see this figure from the appendix of the swe-agent paper:

image
ofirpress commented 2 weeks ago

@klieret thoughts?

klieret commented 2 weeks ago

Hi @ofirpress

The history field contains all input to the model, including the system prompt, the instance prompt and the demonstrations. It den also has the same content as the trajectory, only in a slightly different structure (structured like a chat, rather than combining together action and environment response as in the trajectory field.

Granted, this duplication is not ideal, but it's also what's processed when using convert_traj_to_demo.py etc.

In any way, just renaming the key to demonstrations is equally misleading IMO.

So I vote for 'closing'. Feel free to reopen if you don't agree ;)

ofirpress commented 2 weeks ago

ah ok thanks for clarifying, this makes sense! thank you