Improve Line count criteria

[antonwsu]

I found the following trick that one of the students used to decrease line count:

void treeStop() {
    if (treeFront())
        return stop();
    return move();
}

Clever, but would still be good to deal with it.

[tomitrescak]

That is exactly "non valid solution" and should be marked as invalid. It is against the rules that I have posted. It is not valid java and in the original java compiler it would not compile. You cannot return "void".

[antonwsu]

Agree, but also have to agree that this is a clever exploit. Can only imagine how much time it took this student to find this. Would be good if our compiler would also not allow it, but this is something that can wait.

[tomitrescak]

Fixed for void ... will have to check all other return assignments.

[antonwsu]

This issue is still open. Below is a summary of problems reported by students.

1. By putting function calls inside an if() statement, 1 line of code can be achieved. This method also has the advantage that steps are still counted making the solution appear valid to the step checker, thus putting the score on the leader board. Below is the code for Practical 1- problem 1.

if((turnLeft()

== turnLeft())

==

(move()

== move())

==

(turnLeft()

== turnLeft())

==

(turnLeft()

== move())

){}

2. 0 Lines can be achieved by deleting code as Clara is moving.

3. Leaving the coding windows doesn't terminate the code. This can be demonstrated by running the below code and the pressing the back button in the browser. Refreshing the browser does stop the code executing though.

int x

= 1;

while(x

0){

x++;

__context.message(""

• x,

""

• x);

turnLeft();

}

[tomitrescak]

@antonwsu I have a fool-proof method to solve this once and for all. I can use my transpiler's report to detect how many commands have been used to solve this exercise. Thus, we can then relate to this and number of commands, rather than lines of code.

If you give me a go .. I can fix it, but we will have to re-visit all exercises and adjust the star ratings.

[tomitrescak]

@tomitrescak @sententia @Dean-P @antonwsu let's put our brains together for a more fool proof method of assessing code quality. The Lines of code method is flawed in multiple dimensions and is not even considered to be a really valuable code metric. Instead we can use cyclomatic complexities and and halstead operators.

Check out this library: https://github.com/escomplex/escomplex

For example for the following code:

function m() {
  let m = 1;
}
function f() {
  true ? m() : true ? m() : true ? m() : true ? m() : false;
}

It produces a following report:

{
  "aggregate": {
    "sloc": {
      "logical": 4,
      "physical": 6
    },
    "cyclomatic": 5,
    "halstead": {
      "operators": {
        "distinct": 5,
        "total": 12,
        "identifiers": [
          "function",
          "let",
          "=",
          ":?",
          "()"
        ]
      },
      "operands": {
        "distinct": 5,
        "total": 13,
        "identifiers": [
          "m",
          1,
          "f",
          true,
          false
        ]
      },
      "length": 25,
      "vocabulary": 10,
      "difficulty": 6.5,
      "volume": 83.04820237218406,
      "effort": 539.8133154191964,
      "bugs": 0.027682734124061355,
      "time": 29.989628634399804
    },
    "params": 0,
    "line": 1,
    "cyclomaticDensity": 125
  },
  "functions": [
    {
      "name": "m",
      "sloc": {
        "logical": 1,
        "physical": 3
      },
      "cyclomatic": 1,
      "halstead": {
        "operators": {
          "distinct": 2,
          "total": 2,
          "identifiers": [
            "let",
            "="
          ]
        },
        "operands": {
          "distinct": 2,
          "total": 2,
          "identifiers": [
            "m",
            1
          ]
        },
        "length": 4,
        "vocabulary": 4,
        "difficulty": 1,
        "volume": 8,
        "effort": 8,
        "bugs": 0.0026666666666666666,
        "time": 0.4444444444444444
      },
      "params": 0,
      "line": 1,
      "cyclomaticDensity": 100
    },
    {
      "name": "f",
      "sloc": {
        "logical": 1,
        "physical": 3
      },
      "cyclomatic": 5,
      "halstead": {
        "operators": {
          "distinct": 2,
          "total": 8,
          "identifiers": [
            ":?",
            "()"
          ]
        },
        "operands": {
          "distinct": 3,
          "total": 9,
          "identifiers": [
            true,
            "m",
            false
          ]
        },
        "length": 17,
        "vocabulary": 5,
        "difficulty": 3,
        "volume": 39.47277761308516,
        "effort": 118.41833283925547,
        "bugs": 0.013157592537695053,
        "time": 6.578796268847526
      },
      "params": 0,
      "line": 4,
      "cyclomaticDensity": 500
    }
  ],
  "dependencies": [],
  "maintainability": 156.56646246869644,
  "loc": 1,
  "cyclomatic": 3,
  "effort": 63.209166419627735,
  "params": 0
}

Not the following issues:

1. The cyclomatic complexity of the f method is really high, as well as general cyclomatic complexity
2. There is 5/12 operators and 5/13 operands with total length of 25 (already a good measure of how many components they used)
3. The difficulty, effort, bugs, time and maintainability are also there but are a bit complex to understand.

Any ideas before I present mine ?

[sententia]

My thoughts are we chasing a complexity factor or the ability to write code. For simplistic sake the distinct/used measures is actually quite informative. Especially when we have students who are getting caught nesting everything. The lower the operators and operands the better - they have to think about abuse.

The cyclomatic of 5 makes sense with the number of edges and nodes (nested ifs) in that example, and do we want them to learn this style of trickery. With a looping structure that 5 could be brought down because you reduce the edges and nodes - The density of 125 I'm not too sure about. I thought density was in percentage. the other question is how do you express this so it is meaningful so they know how to fix their code. That then makes me come back to the distinct/used measure of halstead as a better learning metric. My two bob.

[tomitrescak]

Thanks Rhys! Yes, we need to come up with a metric for starting programmers that can help them decide when their code is becoming too complex. As you see the metrics coming from the package we can maybe come with something smart (paper material?)

I agree that it should be informative and easy to understand.

First, we need to understand what makes code good. Often the shortest code is the hardest to maintain, thus it should be a good compromise between length and complexity.

The metrics we can use for this are:

• lines of code
• cyclomatic complexity
• halstead operators and operands

What works for us is that we have 7 years of data in the DB and I can pull up metrics of highest marked code and see the correlations. I'm not sure how relevant the marks actually is as it is a very subjective measure. Do you have any idea on how to spot the good code?

[EDIT] All this can then be used in the UI and we can provide real-time hints whn code is becoming too complex.

[sententia]

I look at the type of decisions. That is hard to measure, because it is an understanding of the problem space. Some students can make some clever decisions in their designs - although they don't get more marks to those who also achieve success, I just acknowledge the design. These designs are usually succinct, and not always complex - just smart and creative. Where they go, yeah I could do it this way, but this makes more sense - creative. I can't see that working for a generic approach. In saying that, those who get top marks they have the same general design with their decisions. Especially the earlier problems

Just thinking outloud - maybe guiding. If we identify the problem 'average' line count before they start they might approach the problem differently, and then we measure the complexity (cyclomatic & halstead) within a range giving feedback after it runs. The range could be an analysis of the previous problems attempts.

Was just looking at halstead effort. ...the mental effort required to develop or maintain a program. The lower a program's Halstead effort, the simpler the program is to change. so how intertwined the solution might be with operators and operands. The word effort is confusing if they don't understand what is happening to get the number.

Thinking outloud again, maybe when combining it's not complexity, but 'learning' difficulty. Especially if you can review commits for certain problems measure the change of effort, complexity until they reach a solution. Then there would be a stat to show the learning difficulty and time required to invest. I don't know just kicking ideas around.