`defect` and `refine` tasks for `codet5p-16b`

[cyw3]

Hi Nice project!

Do you have the checkpoint and evaluation conclusions of the defect and refine tasks for codet5p-16b?

Here I tested the codet5 defect and refine task examples in codetf, and found that the effect is not very good.

[bdqnghi]

hi, thanks for the question. May i know what do you mean by the "effect"? Note that we haven't concluded the project and haven't officially released yet, there are still things to improve.

[cyw3]

• For Salesforce/codet5-base-codexglue-defect:

code_snippets = """
#include <stdio.h>
int main()
{
    int *pointer_var;
    printf("Address of the given pointer variable: %d", pointer_var);
    printf("Value of pointer variable is : %d", * pointer_var);
    return 0;
}
"""
defect_model = load_model_pipeline(model_name="codet5", task="defect",
            model_type="base", is_eval=True,
            load_in_8bit=True, weight_sharding=False)
defects = defect_model.predict([code_snippets])
print(defects)

code_snippets: a c/c++ demo about null pointer error. But Salesforce/codet5-base-codexglue-defect thinks it's not a defect, and output ['false'].

---

• For Salesforce/codet5-base-codexglue-refine-medium:

  code_snippets = """
  class GFG
  {
  public void test ()
  {
      // Initializing String variable with null value
      String ptr = null;
  
      // This line of code throws NullPointerException
      // because ptr is null
      if (ptr.equals("gfg"))
          System.out.print("Same");
      else
          System.out.print("Not Same");
  }
  }
  """
  refine_model = load_model_pipeline(model_name="codet5", task="refine",
          model_type="base", is_eval=True,
          load_in_8bit=True, weight_sharding=False)
  refines = refine_model.predict([code_snippets])
  print(refines)

  code_snippets: a java demo about null pointer error. But Salesforce/codet5-base-codexglue-refine-medium output the fixed version with more defects:

  public void test ( ) {
  // Initializing String variable with null value
  String ptr = null ;
  // This line of code throws NullPointerException because ptr is null
  if ( ptr. equals ( "gfg" ) )
      java.lang.System.out.print ( "Same" ) ;
  else
      java.lang.System.out.print ( "Not Same" ) ;
  }

• defect 1: ptr.equals will have null pointer error.
• defect2: java.lang.System.out.print

[bdqnghi]

i see, thanks for this, the CodeT5-defect is fine-tuned on this data from Microsoft CodeXGLUE: https://github.com/microsoft/CodeXGLUE/tree/main/Code-Code/Defect-detection. In fact, this dataset is mostly used for academic research, apply the results into real-world application can produce unsatisfied result for many cases.

You can check more on the results in CodeT5 paper: https://aclanthology.org/2021.emnlp-main.685.pdf.

[cyw3]

Thanks.

Do you have more datasets about defect and refine to recommend?