LChenGit
commented
3 months ago Hi @chunhualiao and @bin123apple
I have reviewed the log file and summarized the steps in the log file as follows:
Step 1
- prompt: unclear.
- objective: remove comments in Fortran code; get the length of the code
- output:Fortran code without comments.
Step 2
- prompt: unclear. It might be: “”” Please provide a C++ translation of the following Fortran code.
{output of <1>}“””
- objective: get a C++ translation of the given Fortran code
- output: <Fortan, C++> code pairs.
- Observations and Questions:
- what does “qer” mean in line 18: “qer asks for the C++ translation:”?
Step 3
- prompt: “Please provide unit tests to verify the original Fortran and translated C++ codes. Use the specified formats for each language.”
- objective: use LLM to generate unit test
- output:Fortran and C++ code with unit tests
Step 4
- prompt: “Please run these unit tests to verify that the translation from Fortran to C++ has been done correctly.”
- objective: use LLM to run the code with unit tests.
- output: results with “Fortran Stdout”, “Fortran Stderr”, “C++ Stdout” and “C++ Stderr”
- Observations and Questions:
- GPT says in the output: “fortran compile result in 0th turn: Fortran Stdout: Double Precision Test Passed Single Precision Test Passed”. Does GPT actually call the compiler to compile the code? Same question for C++ checking.
- The example output contains “Fortran Stdout”, “Fortran Stderr”, “C++ Stdout” and “C++ Stderr”. Does it consistently contain these four entities in the output? I am curious as it is not specified in the prompt.
Step 5
- prompt: unclear
- objective: fix errors found in step 4 and run the code with the unit test again.
- output: if error repeat or consider the <Fortan, C++> code pair passes the unit test and save them
- Observations and Questions:
- It seems the attempt threshold is 6.
- I noticed that the first example used 6 attempts and still failed to get a pass for a very simple C++ code. Is there a way to improve it in the prompts?
Conclusion
- The log file is complex to interpret as no prompt is given, and the output region is not clear. The prompts listed above are extracted from the log file, but I am not sure if they are the exact prompts used.
- Prompt 4 in Figure 1 of the paper yields True/False returns, which are not found in the log file. Are we still using prompt 4?
- It seems the Compilation and Execution are executted within one prompt. Is it correct? Figure 1 uses two prompts for the two tasks.
- In Figure 1, once the prompt 3 gives a "X", it returns to the LLMs to generate a new unit test case. But in the log, it seems LLMs are fixing the problem, giving the error message instead of generating a new unit test.
- I didn't notice any content related to prompt 6.
My impression is that there are differences.
Please document any differences in this ticket. Thanks.