bin123apple
commented
2 weeks ago I tried token count equals to 300, 600, 900, and 1200. When the token count is larger than 900 and 1200, it is usually hard for the GPT-4 to fix all the bugs during the execution feedback process. But anyway this is parameter that we can adjust.
And for the token count distribution of source dataset. Do you want me to show the distribution for the source dataset or the dataset after deleting comments? Because token count 600 limit is for control the length of the code without comments (dataset_generation/engine_F2C.py line 428-437) :
# Step 1: delete the comments of the source fortran code
fortran_wo_com = delete_comments.format(Fortran_Code = fortran_code)
fortran_wo_com = generate_str_answer_gpt(fortran_wo_com, max_tokens)
fortran_wo_com = fortran_wo_com.encode().decode('unicode_escape','replace')
print(f"fortran_wo_com:\n{fortran_wo_com}")
encoding = tiktoken.encoding_for_model("gpt-4")
token_count = len(encoding.encode(fortran_wo_com,disallowed_special=()))
print("fortran_wo_com length", token_count)
# control the length
if token_count < 600: This step is finished by gpt-4 and we did not save the all code without comments.
chunhualiao
show token count distribution of source dataset.