Randomness in CodeBLEU computation

[iCSawyer]

https://github.com/microsoft/CodeXGLUE/blob/6744a7f6ab658a15382f842df6b9c5f148423a49/Code-Code/code-to-code-trans/evaluator/CodeBLEU/dataflow_match.py#L100

The result of list(set()) is random under some circumstances. It can be easily reproduced by running python -c 'print(list(set(["fa", "dsa", "dsa", "w"])))'.

---

In some cases, the code snippet above can result in the difference of DFG returned by get_data_flow() and cause varying CodeBLUE scores (specifically dataflow_match_score): https://github.com/microsoft/CodeXGLUE/blob/6744a7f6ab658a15382f842df6b9c5f148423a49/Code-Code/code-to-code-trans/evaluator/CodeBLEU/calc_code_bleu.py#L64-L67

---

I have compared these two functions and found that in GraphCodeBERT there is no "merge nodes" action.

https://github.com/microsoft/CodeBERT/blob/ac04c77ca7cda9dc757dc8b4360e358731c8708e/GraphCodeBERT/codesearch/run.py#L68-L104

https://github.com/microsoft/CodeXGLUE/blob/6744a7f6ab658a15382f842df6b9c5f148423a49/Code-Code/code-to-code-trans/evaluator/CodeBLEU/dataflow_match.py#L64-L105

---

My reference and candidate is:

  candidate = \
  '''
  throws IOException {
      int read = super.read(b, off, len);
      if (read > 0) {
          bytesRead.incrementAndGet();
      }
      return read;
  }
  '''
  reference = \
  '''
  throws IOException {
      // Obey InputStream contract.
      checkPositionIndexes(off, off + len, b.length);
      if (len == 0) {
      return 0;
      }

      // The rest of this method implements the process described by the CharsetEncoder javadoc.
      int totalBytesRead = 0;
      boolean doneEncoding = endOfInput;

      DRAINING:
      while (true) {
      // We stay in draining mode until there are no bytes left in the output buffer. Then we go
      // back to encoding/flushing.
      if (draining) {
          to
  '''

I was wondering if #104 ran into the same problem.

Thank you for your replying! @JiyangZhang @Imagist-Shuo @celbree

### Tasks

[ZeyuTeng96]

Hi,

I found the same problem as you addressed above. I think there are two solutions:

1. using list.extend or append func to add nodes
2. adding a sorted() func to it

Do you think is it feasible? We can make a PR to fix this together

[iCSawyer]

Hi,

I found the same problem as you addressed above. I think there are two solutions:

1. using list.extend or append func to add nodes
2. adding a sorted() func to it

Do you think is it feasible? We can make a PR to fix this together

I think your solution could work, and a PR for this is great 🤗

[ZeyuTeng96]

Hi, is it possible to make a PR for it. @celbree