bigcode-project / bigcode-evaluation-harness

A framework for the evaluation of autoregressive code generation language models.
Apache License 2.0
702 stars 180 forks source link

Question regarding stop_at_stop_tokens method #183

Closed phqtuyen closed 1 week ago

phqtuyen commented 5 months ago

Hi all, I have a question regarding the method "stop_at_stop_tokens" implemented here . According to my understanding, this method find the first appearance of a stop_token (which signal the end of a method/function). The code then cut off the generated text after that point. However, would this logic still work with a function/method which call upon helper function/method that is placed after it? For example:

`

    String[] x_split = x.split("/");
    String[] n_split = n.split("/");
    int x_numerator = Integer.parseInt(x_split[0]);
    int x_denominator = Integer.parseInt(x_split[1]);
    int n_numerator = Integer.parseInt(n_split[0]);
    int n_denominator = Integer.parseInt(n_split[1]);
    int lcm = lcm(x_denominator, n_denominator);
    if (lcm % x_denominator == 0 && lcm % n_denominator == 0) {
        int gcd = gcd(x_numerator, n_numerator);
        if (gcd == 1) {
            return false;
        } else {
            return true;
        }
    } else {
        return false;
    }
}

private static int gcd(int a, int b) {
    if (b == 0) {
        return a;
    }
    return gcd(b, a % b);
}

private static int lcm(int a, int b) {
    return a * (b / gcd(a, b));
}

}

`

loubnabnl commented 5 months ago

yes this will truncate helper functions after the main one, but the goal is to generated a self-contained solution, usually if helper functions are needed they're provided in the prompt itself.