Create a list of goal test cases

[evykassirer]

@sangwinc said:

Do you have a comprehensive list of "goal test cases" anywhere in your code, or are they just issues?

This document is based on years of experience from the CAS community: http://www.math.unm.edu/~wester/cas/book/Wester.pdf

Many of these issues only relate to more advanced maths, so aren't (yet) important to you. But, it does contain a lot of careful though about negative powers, roots, zero powers and other issues I notice are included in various other issues.

The answer to this is that it's almost entirely just issues (with the exception of a few tests in the code)

I agree that it'd be great to make a list of goal test cases to guide or development, so here's an issue to coordinate that :) Any ideas for where we'd put it / how it'd be formatted?

[tkosan]

A related question is are the math problems that students submit stored anywhere? If so, it would be useful to analyze these problems so they can be classified. Then maybe test cases could be written for each problem class.

[sangwinc]

I think gathering data on the actual problems solved would be very interesting. I've tended to rely on what is in textbooks, and it would be interesting to see if these are different.

[evykassirer]

Socratic does store these queries! and @aelnaiem is down to make them available for y'all

[aelnaiem]

Yes! I'll share a formatted set of sample questions next week

[aelnaiem]

Here's a list of questions that students have asked us (I've filtered what seem like 'valid' questions). Due to the nature of our application, students learn what types of questions we can explain to them, so unfortunately it's not a very creative list. Students know not to ask Calculus questions yet for example.

But it does still bring up a lot of things we don't support yet including (in order of how common they are in this sample):

• factoring (coming soon #104 )
• system of equations (long term goal)
• negative exponents
• negative roots
• expanding an exponent over parens e.g. (x + 2)^2
• Multi-variate terms e.g. x^2*y^3
• solving equations where x is in the denominator (coming soon #88 )
• fractional exponents
• expanding an exponent over fraction e.g. (2/3)^2
• solving equations when a square root is involved
• absolute with a variable (having to split into multiple cases)
• log/exponentials

sample_questions.txt

[tkosan]

This list of questions is helpful, thanks for posting them.

It would be useful to determine the various ways that elementary algebra expressions can be classified. A number of ways are present in your "not supported yet" list, and more could be identified that are already implemented in mathsteps.

Another way to classify elementary algebra expressions is by their structure. I have been experimenting with enumerating all of the elementary algebra expression trees (which only contain binary operators) to a given depth. The following example shows these trees to depth 3:

http://patternmatics.org/mathsteps/tmp/structure_levels_1_to_3.png

There are 21 depth 3 trees, 651 depth 4 trees, and 457,653 depth 5 trees. I am in the process of creating a LaTeX parser to parse the problems on your list. The first analysis I am going to do on these expressions is to collect statistics on their depths.

[sangwinc]

That is very interesting. Thanks for posting the list of questions.

Is there a reason you have posted these as LaTeX, and not in a "content" form? As Ted says, writing a parser for this set of expressions isn't difficulty (and would be a fun exercise!).

[aelnaiem]

Ahh good question. That's a consequence of how we get student input, but a converted version would be more useful I can imagine. I'll get one :)

[tkosan]

After studying the PRESS research, I suspected that the math problems which are in most textbooks were created using unscientific processes. Discussions such as How do authors make their problems/exercises for their math books? on math.stackexchange support this suspision.

However, logic-based AI software has existed for years that automatically generates math problems using scientific methods:

AGILMAT

This program is based on some amazingly insightful research that the mathematics education system appears to be unaware of.

The opportunity exists for the mathsteps project to expand its scope from solving math problems that were created using unscientific processes to being one of the main sources in the world for obtaining scientifically generated math problems.

[tkosan]

Here is a first attempt to create a parsed version of the list of questions:

sample_questions-1_parse_and_analyze_v.01.txt

The parser is still buggy, so if you notice any errors please point them out.

Each question has an entry that lists the LaTeX version of the question, its line number in the original file, a parsed and printed version that is partially parenthesized, a parsed and printed version that is fully parenthesized, and the maximum depth of the question's expression tree. This histogram shows a distribution of the depths of all the expression trees:

1-2 | *****  | 368
3-4 | ****** | 511
5-6 | *      | 43

I wonder how this distribution correlates with the grade levels of the submitters?

[evykassirer]

I think we could still make a bunch of example tests that we'd like to eventually support - so maybe keep this open?

[aelnaiem]

Hmm, that's fair but I'd rather they be associated with specific issues