search

PyRepair / maniple

8 stars 2 forks source link

Query about Error Information and stack traces #14

Open ramsey-coding opened 6 months ago

ramsey-coding commented 6 months ago

I am unsure about what you are referring to with Error Information and stack traces in this paper.

Error Information: Are you referring to errors in the code generated by the Large Language Model (LLM) that result in syntax errors when you try to compile or execute the code?

Stack Trace: How are stack traces obtained for this project? Do you mean that the code generated by the LLM is causing exceptions, and you are capturing the stack traces when these exceptions are thrown during execution?

Could you please provide more details or clarify these points?

ramsey-coding commented 6 months ago

Also how do you collect runtime value other than instrumentation?

norhh commented 6 months ago

https://github.com/PyRepair/maniple/blob/master/data/BGP32/black/10/110111101/prompt.md You can look at an example prompt here. The error information is the error obtained when executing the test. It consists of the error message which is the AssertionError, and the stacktrace pointing to the location of the error.

norhh commented 6 months ago

The runtime information can be collected by instrumenting the code which stores the values of variables at the start and end of the function and prints it into a file.

ramsey-coding commented 6 months ago

how do you do code instrumentation? Is not it costly? Do you modify the code?

norhh commented 6 months ago

It is cheap for python. It might be a bit expensive for languages like C++ (linking might take some time), but it will be a one time execution. Maniple performs instrumentation by modifying AST.

ramsey-coding commented 6 months ago

Do you share instrumentation code in the artefact?

norhh commented 6 months ago

It is in this repository: https://github.com/PyRepair/pyr_benchmark_wrangling