search

PresConsUIUC / PSAP

Home of the Preservation Self-Assessment Program application.
https://psap.library.illinois.edu/
Other
4 stars 1 forks source link

question dependencies not calculating #351

Closed jhain closed 7 years ago

jhain commented 7 years ago

@adolski, @spelczar and I have been looking at the way questions with dependent questions are calculating and the confirmation is that they're not. As i see it, we have 3 choices, please let me know which you would prefer given the following considerations: 1) fix the math behind the way questions dependencies were outlined - i.e. a dependency question MULTIPLIES (but a fraction of the whole) the resulting score of the parent question that had a dependency. this applied to both the launched PSAP as well as the objects module we're testing. i can give a firm example of what i mean here if that helps, but i assume this is the most time consuming option. 2) change the math behind ALL question dependencies in the live and staged PSAP to fix this by making the scoring work differently by basically making dependent questions no longer depended and asking all questions all the time (will require math changes for calculations to still total). this COULD screw up any previously entered data from folks who had materials with dependent questions though? not sure. 3) change the math as above but only on the PSAP staged site for objects (no permanent data to mess up) and leave the live PSAP alone - knowing that the question dependencies don't work (there aren't a TON of them) but not touching them for now.

thoughts? preferences? JHT

adolski commented 7 years ago

(Pasting @spelczar's original comment below to keep everything together)

The question has to do with the (0.0) weight given to dependent questions. I get why they are weighted as such, because all their "weight" is in the parent question which they are dependent on -- BUT I've done some assessments and math and it seems that that having the dependent questions as (0.0) is actually nullifying whatever people select as the answer to those dependent question - since the answer is multiplied by "0". I tested this on the old (live) PSAP as well as on the staged site and no matter what the answer is to the dependent questions, the score never changes. Am I missing something here, or is this a problem we need to fix?

adolski commented 7 years ago

@jhain @spelczar It is correct that the dependent questions multiply against the parent questions. They always worked that way, AFAIK.

The multiplying could work if the questions were weighed differently... if not being able to achieve a zero score on the question is okay. For example:

Before changes

Results

After changes

Results

jhain commented 7 years ago

@adolski @spelczar Somer and I are sitting here wrapping our heads around this and think i wasn't being clear. what you're stating is correct, but the problem is not in the SCORES being developed in the question dependencies, but in the fact that the WEIGHT of the DEPENDENT questions is blank and therefore calculating as zero (we think). so the question really is, is there a way to have the calcuations work differently if the weight column in the question dependencies is blank (so that the program does not calculate those as an assumed zero and therefore nullify the score) or would it be better for just to re-work our weights to that ALL questions, whether dependent or not have a value in the weight column so that questions are not nullified. i'm happy to chat about this in person if that would help. i'm THINKING that fixing the math so that all quesions have a weight is going to be the way to go, but it will take some thought on the part of somer and I to get the math to work out, so i don't want to go down that path if there's an easy fix on your end.

adolski commented 7 years ago

It seems to me that filling in the weight column for the dependent questions (your option 1) would be the proper way to fix the dependent questions. Then I would just need to modify the code that does the calculations to take the dependent question weights into account.

Once this is done, it will work for all new resource assessments, and I can also update the scores of all existing assessed resources.

spelczar commented 7 years ago

@adolski Quick question about this Alex: You say above that you'd then "need to modify the code that does the calculations to take the dependent question weights into account" - - - which implies that right now the code right now is written to NOT take the weights of the dependent questions into account. But if that is the case, then why is it factoring those weights at "0" (which is what nullifies the parent score). Would there be a way to populate the "weight" column for the dependent questions with a formula that would multiply whatever answer the user selects (ie. yes (0.5), no (0), unsure (0.4)) by the possible points in the parent question? Would that fix the problem? I'm just thinking out loud here. for example: Parent Question: Is the object particularly large, awkward or heavy? Weight 2.5 possible answers: Yes (1), No (1)

1st dependent question: Storage furniture strong enough? possible answers Yes (0.5), No (0), Unsure (0.4)

adolski commented 7 years ago

Sorry, I meant that I will need to modify the code to take the weights into account correctly.

In your example, correct me if I'm wrong, but it looks like the weight of the dependent question would be ignored, and its score would get multiplied into the parent score. That would be fine with me.

Another way to do it would be to fill in the dependent question weights as a percentage of the parent, and make the score for the whole question something like:

question_score = parent_score * (1 - dependent_weight) + dependent_score * dependent_weight

spelczar commented 7 years ago

Alex, Yes - if we can ignore the weight of the dependent question, and multiply the score of each dependent question (which will depend on which answer the user selects) into the parent score. 1-Can we do that? 2- does the fact that most of the OBJECT Questions with dependent questions have TWO dependent questions, not just one, make this any more complicated?

Here is an example:

Parent Question, with 2.5 weight: Is the item large, awkward, heavy? Yes (1), No (1) -if user answers "no" then they get the full 2.5 -if user answers "yes"... then they get sent to the two dependent questions:

Dependent Question #1: Do you have adequate storage furniture for the heavy object? Possible answers: Yes (0.5), No (0), Unsure (0.4)

Dependent Question #2: Do you have adequate staff and/or equipment for the heavy object? Possible answers: staff only (0.25), equipment only (0.25), both (0.5), neither (0)

So, If the user answer to the PARENT question is YES, then they have the POTENTIAL to receive the full 2.5 points based on their answers to the two dependent question in the following way:

Dependent Question #1: if they answer Yes, then they get (0.5) which is 50% of the Yes (1) answer to the parent question. Dependent Question #2: if they answer "Both," then they get (0.5) which is the other 50% of the Yes (1) answer to the parent question. So, by receiving 0.5 for EACH of the child questions, they get 100% of the possible score for the parent question, hence they get the full 2.5 points.

Another Example: If they answer "yes" (0.5) to Dep. Quest. #1, and "staff only" (0.25) for Dep Quest#2, then the answers to those two dependent questions means that the most they could get of the TOTAL possible score of the parent question is 0.75. So based on their answers to the two dependent questions, they would only get 75% of the the possible 2.5 weight for the parent question, so 1.875.

If all this makes sense to you, can we go ahead and rig the "weight column" of the dependent questions so that the "weight" is ignored and instead we take the user-selected answer to the dependent questions and multiply those answers into the parent score.

If this doesn't make sense, we can all meet at the Main library this afternoon to discuss this, just let us know. Thanks!

adolski commented 7 years ago

Just to summarize, here are the options on the table:

1) Using weighted dependent scores without the weight column:

Single dependent: question_score = dependent_score * parent_score Two dependents: question_score = (dependent_score1 * parent_score1) + (dependent_score2 * parent_score2)

2) Using the weight column:

Single dependent: question_score = parent_score * (1 - dependent_weight) + (dependent_score * dependent_weight) Two dependents: question_score = parent_score * (1 - sum_of_dependent_weights) + (dependent_score1 * dependent_weight1) + (dependent_score2 * dependent_weight2)

spelczar commented 7 years ago

Okay, so I'm not clear: does your first option above (Multiplying dependent question scores) match what I was talking about in my github post yesterday?

adolski commented 7 years ago

Sorry, I was skimming too fast and saw the word "multiply" and got the wrong idea. I went back and edited my last post. Option 1 above is now your proposal from yesterday. Option 2 is similar except it doesn't pre-weigh the dependent scores as option 1 does.

Here is one of your dependent question examples using option 1, copied from your post:

Dependent Question #1: Do you have adequate storage furniture for the heavy object? Possible answers: Yes (0.5), No (0), Unsure (0.4)

And here it is adjusted for option 2:

Dependent Question #1 (weight 0.5): Do you have adequate storage furniture for the heavy object? Possible answers: Yes (1), No (0), Unsure (0.8)

spelczar commented 7 years ago

OK Alex, your Option #2 is very similar to a solution that Suhail (grad hourly) came up with, which is essentially: to give the dependent questions weight -- but those weights would ONLY be triggered IF the user even HAD to answer the dependent questions.

Here's what would need change (and I'm forwarding you his email with more detail). Is this in line with your option #2?

  1. Change the %value of the Option which triggers the dependent question to zero.
  2. Add weights to the dependent questions which should be equal parts of the weight of the main question. For example if we have only 1 dependent question then the weight = weight of main question, if we have 2 dependent questions then the weight of each dependent = 50% of weight of main question and so on. (This could be even modified to give more % weight to one dependent question compared to another – ex: 40% 60%)
adolski commented 7 years ago

Change the %value of the Option which triggers the dependent question to zero.

The value of every option, no matter the question, is a number from 0 (worst) to 1 (best). The weights control how much each option affects the whole.

The sum of the dependent question weights must be <= 1. If it's 1, then the value of the parent option that triggers the dependents doesn't matter (because that question's weight will be 0. (1 - sum_of_dependent_weights (1) = 0)

But, if the dependent weights sum to less than 1, then the value of the parent option that triggers the dependents will factor into the score.

Add weights to the dependent questions which should be equal parts of the weight of the main question. For example if we have only 1 dependent question then the weight = weight of main question, if we have 2 dependent questions then the weight of each dependent = 50% of weight of main question and so on. (This could be even modified to give more % weight to one dependent question compared to another – ex: 40% 60%)

Yes, if one dependent question is more important than the other, it can weigh more. And the weights can add up to less than 1.

spelczar commented 7 years ago

@adolski @jhain @suhailsheth If we can't fix any code in the question dependencies for the launch - will pulling out the dependent questions from the object assessment questions to place them on the same level as the parent questions provide a "temporary fix," at least for the NEW PSAP assessment questions?

I guess what I'm asking is: if I redo the QuestionDependencies spreadsheet to make this happen - it shouldn't take any additional coding on your part, should it?

We would just need to 1) make the changes in the spreadsheet; 2) re-ingest the spreadsheet; 3) make sure it works on the staged site; and then 4) we'd be able to launch. Right?

In order to do this, I could potentially be ready for re-ingest of the spreadsheet by end of day today, and then LAUNCH on Monday night.

Is this feasible on your end?

adolski commented 7 years ago

That is all correct, and it will work. The only caveat is that you won't be able to reconfigure the spreadsheet later, to restore the dependent questions once this bug is fixed. Once they go live, they are "locked in," for the most part.

spelczar commented 7 years ago

@adolski OK Alex, we're going to take my last suggestion off the table and go with doing what we can to fix the spreadsheet and then re-ingest and launch.

  1. I will alter the weights in the spreadsheet to coincide with your preferred Option#2 (where the dependency "weights" are a percentage of the parent weight). I can probably have that done by the end of today.

If so, then:

  1. Can we re-ingest the spreadsheet on Monday morning, to make sure that is "works" (as much as it can)? and then
  2. Launch (soft) on Monday evening - so we can test the live product internally and externally. But you will still have the time you need to work on the code fixes for all the question dependencies before the PUBLIC launch (in December I guess).

Hows all this sound?

adolski commented 7 years ago

Sounds good to me.

suhailsheth commented 7 years ago

@adolski @jhain @spelczar

After speaking with Alex on the phone we agreed to go with option 1. We are checking to make sure this will work with all the previous dependent questions and still shooting for ingest on Monday morning and then LAUNCH on Monday night.

OPTION 1: Using weighted dependent scores without the weight column:

Single dependent: question_score = dependent_score * parent_score Two dependents: question_score = (dependent_score1 * parent_score) + (dependent_score2 * parent_score)

spelczar commented 7 years ago

@adolski Alex, the Question Dependencies spreadsheet is up-to-date and ready for one more ingest this evening or Monday morning (before the Launch on Monday PM). The file name is FINAL PSAP Extension Question Dependencies in PSAP > Scores file Thanks.

adolski commented 7 years ago

@spelczar Questions (and everything else) updated on psap-stage.

spelczar commented 7 years ago

THANKS ALEX

adolski commented 7 years ago

I believe this is now working. Examples:

(Scores are rounded to one digit for display.) Please reopen if you notice any problems.

suhailsheth commented 7 years ago

@adolski

For the main question with two dependent questions, the total score is being calculated as question_score = parent_score * dependent_score1 * dependent_score2. This should be question_score = parent_score * (dependent_score1+ dependent_score2) instead.

Example: https://psap-stage.library.illinois.edu/resources/33 assesment score - expected 98.75, Actual 95.6

adolski commented 7 years ago

Thank you @suhailsheth. Here is the updated test list.