Formulagrader debug log

ChristopherChudzicki commented 6 years ago

While working on #26, I wanted a more detailed debug log in FormulaGrader. So...this adds it.

To facilitate testing the debug log, I also added a conftest file that resets random.seed and numpy.random.seed to value None before every test. (Here, None causes system clock to be used.)

Example debug log:

<pre>MITx Grading Library Version 1.0.4<br/>
Student Response:<br/>
z + x*x + f(y)<br/>
<br/>
==============================================================<br/>
FormulaGrader Debug Info<br/>
==============================================================<br/>
Available Functions:<br/>
{   'abs': <ufunc 'absolute'>,<br/>
    'arccos': <function arccos at 0x...>,<br/>
    'arccosh': <ufunc 'arccosh'>,<br/>
    'arccot': <function arccot at 0x...>,<br/>
    'arccoth': <function arccoth at 0x...>,<br/>
    'arccsc': <function arccsc at 0x...>,<br/>
    'arccsch': <function arccsch at 0x...>,<br/>
    'arcsec': <function arcsec at 0x...>,<br/>
    'arcsech': <function arcsech at 0x...>,<br/>
    'arcsin': <function arcsin at 0x...>,<br/>
    'arcsinh': <ufunc 'arcsinh'>,<br/>
    'arctan': <ufunc 'arctan'>,<br/>
    'arctanh': <function arctanh at 0x...>,<br/>
    'conj': <ufunc 'conjugate'>,<br/>
    'cos': <ufunc 'cos'>,<br/>
    'cosh': <ufunc 'cosh'>,<br/>
    'cot': <function cot at 0x...>,<br/>
    'coth': <function coth at 0x...>,<br/>
    'csc': <function csc at 0x...>,<br/>
    'csch': <function csch at 0x...>,<br/>
    'exp': <ufunc 'exp'>,<br/>
    'f': <function f at 0x...>,<br/>
    'fact': <built-in function factorial>,<br/>
    'factorial': <built-in function factorial>,<br/>
    'im': <function <lambda> at 0x...>,<br/>
    'ln': <function log at 0x...>,<br/>
    'log10': <function log10 at 0x...>,<br/>
    'log2': <function log2 at 0x...>,<br/>
    're': <function <lambda> at 0x...>,<br/>
    'sec': <function sec at 0x...>,<br/>
    'sech': <function sech at 0x...>,<br/>
    'sin': <ufunc 'sin'>,<br/>
    'sinh': <ufunc 'sinh'>,<br/>
    'sqrt': <function sqrt at 0x...>,<br/>
    'tan': <ufunc 'tan'>,<br/>
    'tanh': <ufunc 'tanh'>}<br/>
<br/>
<br/>
==========================================<br/>
Evaluation Data for Sample Number 1 of 2<br/>
==========================================<br/>
Variables:<br/>
{   'e': 2.718281828459045,<br/>
    'i': 1j,<br/>
    'j': 1j,<br/>
    'pi': 3.141592653589793,<br/>
    'x': 3.195254015709299,<br/>
    'y': 3.860757465489678,<br/>
    'z': (2.205526752143288+2.0897663659937935j)}<br/>
Student Eval: (14.7111745179+2.08976636599j)<br/>
Compare to:  [(14.711174517877566+2.0897663659937935j)]<br/>
<br/>
<br/>
==========================================<br/>
Evaluation Data for Sample Number 2 of 2<br/>
==========================================<br/>
Variables:<br/>
{   'e': 2.718281828459045,<br/>
    'i': 1j,<br/>
    'j': 1j,<br/>
    'pi': 3.141592653589793,<br/>
    'x': 2.694619197355619,<br/>
    'y': 3.5835764522666245,<br/>
    'z': (1.875174422525385+2.7835460015641598j)}<br/>
Student Eval: (11.9397106851+2.78354600156j)<br/>
Compare to:  [(11.939710685061661+2.7835460015641598j)]<br/>
</pre>

ChristopherChudzicki commented 6 years ago

Oh, run pytest --capture=no to see that conftest does actually run pytest_runtest_setup before every test.

coveralls commented 6 years ago

Coverage remained the same at 100.0% when pulling 4d3b568f7d1f1f775f1afe9ca8954842c585b874 on formulagrader-debug-log into 74619ecb432c6020c805a8840f5320c719ec9c28 on master.

jolyonb commented 6 years ago

Initial thoughts: For each evaluation, after "Student Eval" and "Compare to", can you add a line that says "Equivalency satisfied" or "Not satisfied" or something? Also, the compare to bit will need to be modified for when a comparer_function is supplied; all supplied information should be presented. Finally, the result of that comparison should be given too. Going to go and read the details now.

ChristopherChudzicki commented 6 years ago

I added the comparison function and comparison result to the debug log, and gave the default comparer function a slightly more readable name. I also screwed something up with git, apparently, and so there's some extra commits. Oops.

I don't think there's too much we can provide about the comparison function except its name. But compare_to will be the full comparison parameters (i.e., all of them, not just the first).

<pre>MITx Grading Library Version 1.0.4<br/>
Student Response:<br/>
z + x*x + f(y)<br/>
<br/>
==============================================================<br/>
FormulaGrader Debug Info<br/>
==============================================================<br/>
Available Functions:<br/>
{   'abs': <ufunc 'absolute'>,<br/>
    'arccos': <function arccos at 0x...>,<br/>
    'arccosh': <ufunc 'arccosh'>,<br/>
    'arccot': <function arccot at 0x...>,<br/>
    'arccoth': <function arccoth at 0x...>,<br/>
    'arccsc': <function arccsc at 0x...>,<br/>
    'arccsch': <function arccsch at 0x...>,<br/>
    'arcsec': <function arcsec at 0x...>,<br/>
    'arcsech': <function arcsech at 0x...>,<br/>
    'arcsin': <function arcsin at 0x...>,<br/>
    'arcsinh': <ufunc 'arcsinh'>,<br/>
    'arctan': <ufunc 'arctan'>,<br/>
    'arctanh': <function arctanh at 0x...>,<br/>
    'conj': <ufunc 'conjugate'>,<br/>
    'cos': <ufunc 'cos'>,<br/>
    'cosh': <ufunc 'cosh'>,<br/>
    'cot': <function cot at 0x...>,<br/>
    'coth': <function coth at 0x...>,<br/>
    'csc': <function csc at 0x...>,<br/>
    'csch': <function csch at 0x...>,<br/>
    'exp': <ufunc 'exp'>,<br/>
    'f': <function f at 0x...>,<br/>
    'fact': <built-in function factorial>,<br/>
    'factorial': <built-in function factorial>,<br/>
    'im': <function <lambda> at 0x...>,<br/>
    'ln': <function log at 0x...>,<br/>
    'log10': <function log10 at 0x...>,<br/>
    'log2': <function log2 at 0x...>,<br/>
    're': <function <lambda> at 0x...>,<br/>
    'sec': <function sec at 0x...>,<br/>
    'sech': <function sech at 0x...>,<br/>
    'sin': <ufunc 'sin'>,<br/>
    'sinh': <ufunc 'sinh'>,<br/>
    'sqrt': <function sqrt at 0x...>,<br/>
    'tan': <ufunc 'tan'>,<br/>
    'tanh': <ufunc 'tanh'>}<br/>
<br/>
<br/>
==========================================<br/>
Evaluation Data for Sample Number 1 of 2<br/>
==========================================<br/>
Variables:<br/>
{   'e': 2.718281828459045,<br/>
    'i': 1j,<br/>
    'j': 1j,<br/>
    'pi': 3.141592653589793,<br/>
    'x': 1.9792909070108546,<br/>
    'y': 4.216314218606578,<br/>
    'z': (1.0236442393816596+2.6483563171032602j)}<br/>
Student Eval: (6.75746487613+2.6483563171j)<br/>
Compare to:  [(6.7574648761254101+2.6483563171032602j)]<br/>
Comparer Function: <function default_equality_comparer at 0x...><br/>
Comparison Satisfied: True<br/>
<br/>
<br/>
==========================================<br/>
Evaluation Data for Sample Number 2 of 2<br/>
==========================================<br/>
Variables:<br/>
{   'e': 2.718281828459045,<br/>
    'i': 1j,<br/>
    'j': 1j,<br/>
    'pi': 3.141592653589793,<br/>
    'x': 1.97893195498433,<br/>
    'y': 2.9832068192975476,<br/>
    'z': (1.2755923633979924+2.1390522458319143j)}<br/>
Student Eval: (8.10002232035+2.13905224583j)<br/>
Compare to:  [(8.1000223203532205+2.1390522458319143j)]<br/>
Comparer Function: <function default_equality_comparer at 0x...><br/>
Comparison Satisfied: True<br/>
</pre>

jolyonb commented 6 years ago

Ok, can you do an interactive rebase to squash all these commits into one? It got a bit messy!

https://github.com/edx/edx-platform/wiki/How-to-Rebase-a-Pull-Request

ChristopherChudzicki commented 6 years ago

@jolyonb done

jolyonb commented 6 years ago

LGTM 👍

mitodl / mitx-grading-library

Formulagrader debug log #37