search

langcog / web-cdi

7 stars 5 forks source link

extend norms for English CAT to 36 months #509

Closed vmarchman closed 3 months ago

vmarchman commented 3 months ago

@HenryMehta We have figured out a way to have the norms for the English CAT extend to 36 months. I think that all is involved is:

(1) extend maximum benchmark age from 30 to 36 months. So, all folks up to 36 months will be compared to their actual (or corrected) age, and all folks older than 36 months will be compared to 36 months.

(2) @kachergis put the tables for the Theta look ups in slack but the link is here: [https://rpubs.com/kachergis/CDI-CAT-EN-norms]

(3) here are the tables for the vocabulary conversion, i.e., to look up what vocabulary score is equivalent to a particular theta:

WSprod_both_36months.csv WSprod_boys_36months.csv WSprod_girls_36months.csv

Nothing about the WS or WG norms will change or the Spanish CAT will change.

Let me know if you have any questions!

HenryMehta commented 3 months ago

@vmarchman The girsl file only shows to 30 months

HenryMehta commented 3 months ago

Also, the CAT lookup uses the WS norms so I have to add these to the WS files, which means the WS will find them (I think). And since it only has Total Produced, that might cause a problem. I'll implement locally and test

HenryMehta commented 3 months ago

@vmarchman I've deployed to dev (without girls numbers at this point). It looks to me like it is working for CAT. For WS I believe it is getting the 31-36 month figure for total produced but using the 30 month figure for the other categories.

If you want to stop this happening, I will not to amend how the CAT benchmarks equivalents are derived by the system which will take 3-4 hours I believe. Let me know

vmarchman commented 3 months ago

@HenryMehta Ok - sorry - Here's the corrected girls file. WSprod_girls_36months.csv

At this point, we want the WS norms to remain the same. The extension out to 36 months will only happen with the CAT. I don't want to extend the WS because, as you say, it's only the production and the other variables will be out to 30 months, so the "benchmark" age will not apply consistently across the variables.

Is it possible to use one set of files for the CAT and another for the WS?

HenryMehta commented 3 months ago

@vmarchman I have amended and loaded. I believe you should now get the right benchmarks upto 36 months for CAT, and only up to 30 months for WS (or WG, I cannot remember which).

I will be doing some more testing later, but it is good for you to test now as well

kachergis commented 3 months ago

Great, thanks Henry! Do you have experience writing automated tests (e.g. using unittest*)? WebCDI seems like a great candidate for tests... Probably well worth setting up, and I'm sure I and others could help write tests if needed.

*https://docs.djangoproject.com/en/5.0/topics/testing/

On Tue, Mar 19, 2024 at 4:10 AM Henry Mehta @.***> wrote:

@vmarchman https://github.com/vmarchman I have amended and loaded. I believe you should now get the right benchmarks upto 36 months for CAT, and only up to 30 months for WS (or WG, I cannot remember which).

I will be doing some more testing later, but it is good for you to test now as well

— Reply to this email directly, view it on GitHub https://github.com/langcog/web-cdi/issues/509#issuecomment-2006880807, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAVUWP3K44FTAOLGIZ3COHLYZAMLFAVCNFSM6AAAAABEYWDTU6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBWHA4DAOBQG4 . You are receiving this because you were mentioned.Message ID: @.***>

HenryMehta commented 3 months ago

@vmarchman I can write unit tests for the site, but this shouldn't remove the need to user test and ensure the process is working correctly. A unit test will do something like, create a study for American WS (and if we want to test every instrument we write a test for every instrument). But it won't test the actual user experience. But I can certainly write them.

kachergis commented 3 months ago

agreed, still also necessary to test manually!

On Tue, Mar 19, 2024 at 12:41 PM Henry Mehta @.***> wrote:

@vmarchman https://github.com/vmarchman I can write unit tests for the site, but this shouldn't remove the need to user test and ensure the process is working correctly. A unit test will do something like, create a study for American WS (and if we want to test every instrument we write a test for every instrument). But it won't test the actual user experience. But I can certainly write them.

— Reply to this email directly, view it on GitHub https://github.com/langcog/web-cdi/issues/509#issuecomment-2007983043, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAVUWP3XOSOMZVHR5RWYRQ3YZCIFTAVCNFSM6AAAAABEYWDTU6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBXHE4DGMBUGM . You are receiving this because you were mentioned.Message ID: @.***>

vmarchman commented 3 months ago

@Henry @.***> Sorry for the late reply. re building the tests: Let me discuss at the CDI meeting tomorrow.

I'll try to manually test CAT today and get back to you asap.

Virginia

On Tue, Mar 19, 2024 at 2:45 PM George Kachergis @.***> wrote:

agreed, still also necessary to test manually!

On Tue, Mar 19, 2024 at 12:41 PM Henry Mehta @.***> wrote:

@vmarchman https://github.com/vmarchman I can write unit tests for the site, but this shouldn't remove the need to user test and ensure the process is working correctly. A unit test will do something like, create a study for American WS (and if we want to test every instrument we write a test for every instrument). But it won't test the actual user experience. But I can certainly write them.

— Reply to this email directly, view it on GitHub https://github.com/langcog/web-cdi/issues/509#issuecomment-2007983043,

or unsubscribe < https://github.com/notifications/unsubscribe-auth/AAVUWP3XOSOMZVHR5RWYRQ3YZCIFTAVCNFSM6AAAAABEYWDTU6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBXHE4DGMBUGM>

. You are receiving this because you were mentioned.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/langcog/web-cdi/issues/509#issuecomment-2008186564, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB2TUTEPKHG6N7ZO23KYLULYZCWV7AVCNFSM6AAAAABEYWDTU6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBYGE4DMNJWGQ . You are receiving this because you were mentioned.Message ID: @.***>

vmarchman commented 3 months ago

Hi @Henry @.***> I've tested CAT and things look good now.

re tests. If you think that they are useful, @George Kachergis @.***> makes a good case for them. If you want to work on some as we go along, that's fine, but only if we don't have other priorities. I understand that there will still be the need for "real" user testing (i.e., me) :-)

Best,

Virginia

On Wed, Mar 20, 2024 at 8:41 AM Virginia Marchman @.***> wrote:

@Henry @.***> Sorry for the late reply. re building the tests: Let me discuss at the CDI meeting tomorrow.

I'll try to manually test CAT today and get back to you asap.

Virginia

On Tue, Mar 19, 2024 at 2:45 PM George Kachergis @.***> wrote:

agreed, still also necessary to test manually!

On Tue, Mar 19, 2024 at 12:41 PM Henry Mehta @.***> wrote:

@vmarchman https://github.com/vmarchman I can write unit tests for the site, but this shouldn't remove the need to user test and ensure the process is working correctly. A unit test will do something like, create a study for American WS (and if we want to test every instrument we write a test for every instrument). But it won't test the actual user experience. But I can certainly write them.

— Reply to this email directly, view it on GitHub https://github.com/langcog/web-cdi/issues/509#issuecomment-2007983043,

or unsubscribe < https://github.com/notifications/unsubscribe-auth/AAVUWP3XOSOMZVHR5RWYRQ3YZCIFTAVCNFSM6AAAAABEYWDTU6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBXHE4DGMBUGM>

. You are receiving this because you were mentioned.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/langcog/web-cdi/issues/509#issuecomment-2008186564, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB2TUTEPKHG6N7ZO23KYLULYZCWV7AVCNFSM6AAAAABEYWDTU6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBYGE4DMNJWGQ . You are receiving this because you were mentioned.Message ID: @.***>

HenryMehta commented 3 months ago

Hi @vmarchman, yes, the can be useful. I'll work on a few

HenryMehta commented 3 months ago

@vmarchman CAT changes a live. I'll close this and open a test script ticket

vmarchman commented 3 months ago

Thank you @Henry @.***> !

On Fri, Mar 22, 2024 at 9:05 AM Henry Mehta @.***> wrote:

@vmarchman https://github.com/vmarchman CAT changes a live. I'll close this and open a test script ticket

— Reply to this email directly, view it on GitHub https://github.com/langcog/web-cdi/issues/509#issuecomment-2015416025, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB2TUTHDCJPQCXK4UTMSWJDYZRJGJAVCNFSM6AAAAABEYWDTU6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJVGQYTMMBSGU . You are receiving this because you were mentioned.Message ID: @.***>