Closed AlexSwiderski closed 2 years ago
I think the thing to do would be to compare the ability estimates and sem after each response between pnt-cat and catpuccino. If those are also different perhaps something is wrong.
I think the pnt cat estimates are based on a larger sample of pwa than catpuccino though right? Could that potentially change the item difficulty estimates?
If this is the case, then when Will sends me the scores from his revised estimates using the Stan code, I can check out app to make sure the estimates are right.
Some quick thoughts: Up until I sent Rob the new T-scaled difficulties in the last couple of weeks, the difficulty estimates should have been the same as the ones we used in the R03. Also, I'm pretty sure the item selection (Max Fisher Info) was the same in both cases. That said, there might have slight differences in the other EAP scoring parameters that were different. With the update to the new T-scaled parameters, I'd be surprised in the difficulty estimates didn't move around relative to each other at least a little, because it's a larger sample. Given that there's 90% overlap here, that seems like the most parsimonious explanation for this behavior.
From: Alex Swiderski @.> Sent: Monday, October 18, 2021 12:49 PM To: aphasia-apps/pnt @.> Cc: Hula, William D @.>; Mention @.> Subject: [EXTERNAL] [aphasia-apps/pnt] Difficulty estimates (Issue #31)
It occurred to me this morning that the pnt-cat30 is delivering items differently than the pnt-cat30 when the participant gets the first couple correct, such that microscope is the third item delivered.
The order when a participant gets all items correct from the anomia R03 for the first 10 items is:
pumpkin, ruler, cheerleaders, volcano, microscope, stethoscope, pyramid, binoculars, dinosaur, and ambulance.
However, in the PNT-CAT they are: pumpkin, ambulance, microscope, stethoscope, pyramid, binoculars, volcano, dinosaur, cheerleaders, thermometer.
Just making a note of this because it could mean a couple things:
@william-hulahttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fwilliam-hula&data=04%7C01%7C%7C8538ee4f408b4272a1a808d992573656%7Ce95f1b23abaf45ee821db7ab251ab3bf%7C0%7C0%7C637701725539388638%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=02OCdcjcFrYFJwfbEPoEtZArSleTHjjUzW7jhaiKRrs%3D&reserved=0 @rbcavanaughhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Frbcavanaugh&data=04%7C01%7C%7C8538ee4f408b4272a1a808d992573656%7Ce95f1b23abaf45ee821db7ab251ab3bf%7C0%7C0%7C637701725539398595%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=c9q0Nbgq1%2FGI%2B73VB65gBD0NGGg7szdjh7367H%2FcsRA%3D&reserved=0
- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Faphasia-apps%2Fpnt%2Fissues%2F31&data=04%7C01%7C%7C8538ee4f408b4272a1a808d992573656%7Ce95f1b23abaf45ee821db7ab251ab3bf%7C0%7C0%7C637701725539398595%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=tI%2FxeeDVWbbCO8cgPqZO6dtlzJ3tB%2ByLqg2V6yv1oe0%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAU44R7DIQOYKDPVQ4XDNQY3UHRFYLANCNFSM5GHDLI4Q&data=04%7C01%7C%7C8538ee4f408b4272a1a808d992573656%7Ce95f1b23abaf45ee821db7ab251ab3bf%7C0%7C0%7C637701725539408552%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yjItB5Lo%2Bi1A9i2rkUUUlbviJkidbo7K53IShvzSpUk%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7C%7C8538ee4f408b4272a1a808d992573656%7Ce95f1b23abaf45ee821db7ab251ab3bf%7C0%7C0%7C637701725539418508%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=rHsL9QSGwEN43TRbnOFtkXe0ntZpvrjHDPopqmt6c%2BE%3D&reserved=0 or Androidhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7C%7C8538ee4f408b4272a1a808d992573656%7Ce95f1b23abaf45ee821db7ab251ab3bf%7C0%7C0%7C637701725539428464%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=UpQQcN07G4oQyBZwz0OAyiGK%2Fo%2FoGgZf3F6%2FiZlCcMA%3D&reserved=0.
It occurred to me this morning that the pnt-cat30 is delivering items differently than the pnt-cat30 when the participant gets the first couple correct, such that microscope is the third item delivered.
The order when a participant gets all items correct from the anomia R03 for the first 10 items is:
pumpkin, ruler, cheerleaders, volcano, microscope, stethoscope, pyramid, binoculars, dinosaur, and ambulance.
However, in the PNT-CAT they are: pumpkin, ambulance, microscope, stethoscope, pyramid, binoculars, volcano, dinosaur, cheerleaders, thermometer.
Just making a note of this because it could mean a couple things: 1) are item choice algorithm is different than catpuchino 2) our item difficulty parameters have changed 3) An error occurred somewhere in the t-score transformation 4) something else.
@william-hula @rbcavanaugh