search

openedx-unsupported / ease

EASE (Enhanced AI Scoring Engine) is a library that allows for machine learning based classification of textual content. This is useful for tasks such as scoring student essays.
GNU Affero General Public License v3.0
216 stars 96 forks source link

possible enhancement - decoupling discern from ease #33

Closed kern3020 closed 11 years ago

kern3020 commented 11 years ago

Hello,

Most programmers will be interacting with ease through discern’s REST API. They will report bugs based on the REST API. ease is the special sauce which differentiates edX. At the moment, it is difficult to debug ease problems because it is intertwined with celery. What if we created a mechanism to translate the REST API into an ease specific problems? Decoupling it from discern means the engineer can focus on machine learning and won’t have to concern oneself with the web, db or queue. Also, we can add it to the ease testsuite. What do you think?

-jk

VikParuchuri commented 11 years ago

I'm not sure what you mean. ease here is a library that is exposing two high level functions (grade and create) that discern is using. You could create a debug mode in discern (and indeed, the test_settings there are a start) where you moved celery out of the loop (CELERY_ALWAYS_EAGER sort of does this), but I'm not sure how much more you could decouple two elements that are already very decoupled.

Do you have a specific idea in mind?

kern3020 commented 11 years ago

Hello,

What if any call to ease would dump the input of the ease functions to disk if a specific flag/option/variable was set. Other then this functionality, discern/ease would be unchanged. Perhaps, these inputs might be represented as xml on disk. Code would be added to the ease testsuite to read the input from the files we just created. At this point, it would be trival to create a test as new bugs are reported.

Imagine it is 6 months or a year from now. You and this projects are widely successful. Many courses rely on discern/ease to grade essays. Consider the follow two scenarios with the existing source verses one augmented with this change.

1) The most important customer reports a hot bug at the same time you are working on an important new feature. Let's assume we get a testcase based on discerns REST API. [aside: I just made up the Tom Sawyer score. How many people can help the core developer? ]

existing, no change: A support person can confirm the problem. With the existing source base, a core developer who knows both discern (ie, web service, database and queues) and ease(machine learning) would be needed to debug it. Once this key developer has slain the bug, it is my experience that a test will not be writen because there is no systematic way to extract the input and the developer is tired.

Tom Sawyer score: 1

With the change: A support person can confirm the problem. The bug needs to be isolated. In this case, a web developer confirms there is no problem with discern and dumps the input for ease. We still need a core developer but they only need to know ease. If the web developer also nows machine learning well, go for it. What if their is an engineer who focuses exclusively on machine learning? They have a nice starting point too. When the bug is closed, a QA person has the data then need to develop a good test.

Tom Sawyer score: 3

2) This is the same as number one except it is the second time they reported the problem. You just fixed this 4 months ago. Yikes!

existing, no change: The approach is the same is number 1. You have the advantage of either remembering the bug or looking it up in the bug database. However, the institution reporting the bug again is sad.

with the change: QA likely added the testcase to the testsuite. this regression is significantly less likely to happen.

-jk

VikParuchuri commented 11 years ago

I like this, thanks.

Steps I can/will take:

VikParuchuri commented 11 years ago

Done with https://github.com/edx/ease/pull/39 . You can set the dump_data = True flag in create.create. After doing this, it will create a json file in the data/json_data directory. From there on out (unless you delete the file), whenever you run nosetests (can use sh run_tests.sh), it will run the test case. You can repeat the data dump process as many times as you want...it will run all the tests on each of the data files.

kern3020 commented 11 years ago

Excellent. I want to try it immediately! Alas, it will have to wait until tomorrow.

Thanks, John