search

RunestoneInteractive / RunestoneComponents

Packaging of the Runestone tools for publishing educational materials using github pages
http://runestoneinteractive.org
Other
101 stars 225 forks source link

Problem with new A/B testing ability. #1122

Closed barbarer closed 3 years ago

barbarer commented 3 years ago

I tried the new A/B option on the ..selectquestion directive with an option of :fromid: A, B and expected it to pick either A or B randomly and give me all the questions from A or B. However, it seems to have mixed in some from A and some from B. See https://runestone.academy/runestone/books/published/py4e-int/exp1/inst1.html. The A items should not have distractors and the B ones should. I am seeing 3 problems with distractors and one without.

There are several questions on one page like this:

.. selectquestion:: sum13_selq :ab: disPPEx1 :fromid: sum13_nd_pp_exA, sum13_w6d_pp_exB

bnmnetp commented 3 years ago

You had three entries in the experiment table two for group A one for group B. Not 100% sure why you ended up with duplicates, but will sort the entries to make sure you are consistently in the first group.

To be clear I'm not sure why you ended up with a second entry but after the second entry I understand why it was going to keep adding. Should be fixed soon.

barbarer commented 3 years ago

I have run the experiment several times. That could be why there are several entries.

Dr. Barbara Ericson Assistant Professor, School of Information University of Michigan

On Mon, Jan 18, 2021 at 1:25 PM Bradley Miller notifications@github.com wrote:

You had three entries in the experiment table two for group A one for group B. Not 100% sure why you ended up with duplicates, but will sort the entries to make sure you are consistently in the first group.

To be clear I'm not sure why you ended up with a second entry but after the second entry I understand why it was going to keep adding. Should be fixed soon.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/RunestoneInteractive/RunestoneComponents/issues/1122#issuecomment-762408795, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKOZ7M3ZKSPNV43F5WHHHXTS2R4JDANCNFSM4WHRN7IA .

bnmnetp commented 3 years ago

You are not using the AB questions in a timed exam. That was the use case that you described.

Since you are using A/B questions on a regular page, all of the requests to select the questions are sent (essentially) simultaneously. With multiple processors that creates a condition where two or more requests are handled in parallel, and neither of them sees that you have been placed in a group so both of them randomly pick a group. (bad luck if you get both groups selected)

Inside. timed exam the questions are instantiated sequentially as the student navigates through the questions. So no race condition.

The solution is to figure out if we can serialize the creation of the selectquestions outside of a timed exam. In the past it has been safe to create all the components asynchronously to speed up page rendering.

This is going to require some research.

barbarer commented 3 years ago

I can just put all of these questions in a timed exam for now.

bnmnetp commented 3 years ago

Give me a day to research this and see if there is a better solution.

Someone else will surely decide to do the same thing and then it will fail for them too.

barbarer commented 3 years ago

Okay.

Dr. Barbara Ericson Assistant Professor, School of Information University of Michigan

On Tue, Jan 19, 2021 at 7:43 PM Bradley Miller notifications@github.com wrote:

Give me a day to research this and see if there is a better solution.

Someone else will surely decide to do the same thing and then it will fail for them too.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/RunestoneInteractive/RunestoneComponents/issues/1122#issuecomment-763245179, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKOZ7M54FIISBRUEETCKOQTS2YRMLANCNFSM4WHRN7IA .

bnmnetp commented 3 years ago

Good news. A bit of research and this didn't turn out to be too hard.

All that work on moving the internals to async/await have paid off in this case.

I'll have to do a new release of runestone components, so after Saturday you should be good to go.

bnmnetp commented 3 years ago

I built py4e-int on dev.runestoneinteractive.org and have tested my fix to the experiment there. It all seems to work fine for me, but if you want to try it there, that would be great.