CAT scoring

[vmarchman]

@HenryMehta

(1) When administering CAT to children over 30 months of age, the output for benchmarks/lookups does not populate in the csv files. It only populates in the clinical reports. The benchmark values go to 36 months, so it should be able to do the lookup.

(2) For children who are older than 36 months, I originally got a Server error when I tried to download the clinical report for that individual child. I just tried again, and the clinical report now gives a blank pdf file. When I download the csv, I get the output, but the benchmark values are blank.

I think the problem is that there is no "age to compare" in the csv outputs.

[HenryMehta]

Hi @vmarchman, I hope the move has gone well. Do you want me to pick this up asap/next?

[vmarchman]

That would be great - thanks!

On Tue, Feb 20, 2024 at 9:35 AM Henry Mehta @.***> wrote:

Hi @vmarchman https://github.com/vmarchman, I hope the move has gone well. Do you want me to pick this up asap?

— Reply to this email directly, view it on GitHub https://github.com/langcog/web-cdi/issues/507#issuecomment-1954711581, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB2TUTDNLTJ67KJ6EK5VJ5TYUTNGBAVCNFSM6AAAAABDRR5APSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJUG4YTCNJYGE . You are receiving this because you were mentioned.Message ID: @.***>

[HenryMehta]

@vmarchman I've just checked. The benchmarks only go to 30 months so I've amended the csv downloads so that if child age is greater than max age in benchmark data, then use the max age. I've deployed this to testing.webcdi.org and tested there and seems to be working to me. Please check you're happy and let me know and I'll deploy to production.

I've also noticed that if the age is greater than 36 then the administration won't start but errors because we don't have an initial word from CAT. I could do something about this if you want (eg if age > 36 use initial word for 36 months). Please let me know if you would like me to do this

[HenryMehta]

@vmarchman Actually, I've done an update so if age is greater than 36 the starting word for the administration will be for a child aged 36 months. Let me know if I need to undo this

[vmarchman]

@Henry @.***> Yes - looks great. Agree that we use the starting word for 36 months for all kids > 36 months.

Re the lookup percentiles. For kids > 30 months, the adjusted age to compare in the clinical report and in the csv outputs should be that max age, not their chronological age.

But, are there two age ranges? It goes to 36 months for the theta percentiles, but only 30 months for the vocab look ups?

On Wed, Feb 21, 2024 at 1:11 AM Henry Mehta @.***> wrote:

@vmarchman https://github.com/vmarchman Actually, I've done an update so if age is greater than 36 the starting word for the administration will be for a child aged 36 months. Let me know if I need to undo this

— Reply to this email directly, view it on GitHub https://github.com/langcog/web-cdi/issues/507#issuecomment-1956198530, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB2TUTDQOCMV3K4W7RA3QBTYUW24TAVCNFSM6AAAAABDRR5APSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJWGE4TQNJTGA . You are receiving this because you were mentioned.Message ID: @.***>

[HenryMehta]

@vmarchman I think it only going to 30 for both. I don't have a lookup that goes to 36

[vmarchman]

Ok - So then the age to compare is always max = 30 months.

On Wed, Feb 21, 2024 at 8:14 AM Henry Mehta @.***> wrote:

@vmarchman https://github.com/vmarchman I think it only going to 30 for both. I don't have a lookup that goes to 36

— Reply to this email directly, view it on GitHub https://github.com/langcog/web-cdi/issues/507#issuecomment-1957153626, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB2TUTC5AS4WAYZARUCEE7LYUYMQBAVCNFSM6AAAAABDRR5APSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJXGE2TGNRSGY . You are receiving this because you were mentioned.Message ID: @.***>

[HenryMehta]

@vmarchman Yes, that is right. I'm not clear. Are you asking me to change something or is it good as is?

[vmarchman]

Hi @Henry @.***> Sorry. Yes, there should be another column in the csv output, like for the other forms, representing "age to compare" or "benchmark age". It should go right before the percentiles. For all children who are between 12 and 36 months, the value in the age and the benchmark age column will be the same. But for children > 30 months, the benchmark age column should be 30 months (since that is the highest age for the lookup tables) whereas their "age" column reflects their actual, chronological age.

This value should also go in the clinical reports in the "Age to compare" box.

Also, the benchmark age should be different in the cases when children have "adjusted benchmarks" due to prematurity, like we do for the other forms. Was that set up for the CAT?

On Wed, Feb 21, 2024 at 10:34 PM Henry Mehta @.***> wrote:

@vmarchman https://github.com/vmarchman Yes, that is right. I'm not clear. Are you asking me to change something or is it good as is?

— Reply to this email directly, view it on GitHub https://github.com/langcog/web-cdi/issues/507#issuecomment-1958793191, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB2TUTG6PL6C7HBHTEHI6NLYU3RIBAVCNFSM6AAAAABDRR5APSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJYG44TGMJZGE . You are receiving this because you were mentioned.Message ID: @.***>

[HenryMehta]

@vmarchman ok, I'll work through it tomorrow

[vmarchman]

Hi @Henry @.> @George Kachergis @.> George is going to give you new lookup tables for the theta values to 36 months (now only going to 30 months).

So, for the theta percentiles, the age and age to compare would be the same up to 36 months, and then > 36 months, the age to compare remain 36 months.

For the vocabulary estimates, for kids up to 30 months, the age and age to compare would be the same. After 30 months, the vocabulary estimates are blank (since we don't have vocabulary scores after that age).

We may in the future try to figure out making new vocabulary tables that go up to 36 months.

Let me know if you have any questions!

-vm

On Thu, Feb 22, 2024 at 7:36 AM Henry Mehta @.***> wrote:

@vmarchman https://github.com/vmarchman ok, I'll work through it tomorrow

— Reply to this email directly, view it on GitHub https://github.com/langcog/web-cdi/issues/507#issuecomment-1959705912, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB2TUTCAUQCFKNUMVWEQGP3YU5QY5AVCNFSM6AAAAABDRR5APSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJZG4YDKOJRGI . You are receiving this because you were mentioned.Message ID: @.***>

[HenryMehta]

@vmarchman This will be easier for you to check what I've done with only upto 30 months loaded.

For CAT I've added a new column/field called Benchmark Cohort Age. This is the age group used in benchmark data. I have left Age to Compare (which is the age adjusted for being born early/late). This allows the researcher to know they want to compare the Age to Compare for a certain age, but if we don't have the data they're actually getting a different age and this is it. I hope that makes sense.

Before I apply the same principle to other instruments, could you take a look and decide if you like this or if you want me to remove it and simply apply the Benchmark Cohort Age in the Age to Compare field.

[vmarchman]

Hi @Henry @.***> - Yes I see how that works. Thanks!

But, let's change the terminology a bit to be more clear. Let's rename "Age to Compare" to "Adjusted Age". We can keep "Benchmark Cohort Age".

Ok, this is how it should work across all forms then. I just tested the English WS with a 38 month old kid. The Age to Compare says 38, but it should be 30 months and the naming/variables should be parallel to the CAT.

Thanks!!!

Virginia

On Fri, Feb 23, 2024 at 12:40 AM Henry Mehta @.***> wrote:

@vmarchman https://github.com/vmarchman This will be easier for you to check what I've done with only upto 30 months loaded.

For CAT I've added a new column/field called Benchmark Cohort Age. This is the age group used in benchmark data. I have left Age to Compare (which is the age adjusted for being born early/late). This allows the researcher to know they want to compare the Age to Compare for a certain age, but if we don't have the data they're actually getting a different age and this is it. I hope that makes sense.

— Reply to this email directly, view it on GitHub https://github.com/langcog/web-cdi/issues/507#issuecomment-1960925315, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB2TUTE2PJOKEMVHJM2CKJDYVBIXFAVCNFSM6AAAAABDRR5APSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRQHEZDKMZRGU . You are receiving this because you were mentioned.Message ID: @.***>

[HenryMehta]

@vmarchman Great. I've only applied to English CAT at this point. I'll apply to others over weekend or early next week. I'm going to be away from Thursday 29th for 1 week so let's try and deploy to production before then

[HenryMehta]

@vmarchman all available to test now

[vmarchman]

Hi @Henry @.***> I tested CAT and English WG and WS.

CAT seems to be ok.

WG and WS have issues with Adjusted vs. Benchmarking ages.

(1) WG csv is ok, but clinical report has wrong values in Adjusted Age and Benchmark age. Not sure what is happening in Adjusted age.

(2) WS Clinical report seems to be ok, but csv has wrong values. The csv has Adjusted age = Benchmark age, but it should be same as Chron Age.

Note that for Adjusted Age, the number of weeks premature is subtracted from the Age (chronological age). The benchmark age is the same as the Adjusted age for those children within the age range of the norms (i.e., 8-18 or 16-30). So, a child who is 16 months and 8 weeks premature would have Age = 16, Adjusted Age = 14, and Benchmark age = 14.

If they are older than the age range and premature, the adjusted age is the same as the chron age minus # of weeks premature, but the benchmark age is the oldest possible. That is, don't subtract the number of months to adjust from the oldest age in those cases. So, a child who is 36 weeks old and 8 weeks premature, would have Age = 36, Adjusted age = 34, and Benchmark age = 30.

I can do some more testing tomorrow!

-vm

On Mon, Feb 26, 2024 at 3:55 AM Henry Mehta @.***> wrote:

@vmarchman https://github.com/vmarchman all available to test now

— Reply to this email directly, view it on GitHub https://github.com/langcog/web-cdi/issues/507#issuecomment-1963971768, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB2TUTC3ZVYBRQWT4JLWWOLYVRZ3BAVCNFSM6AAAAABDRR5APSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRTHE3TCNZWHA . You are receiving this because you were mentioned.Message ID: @.***>

[HenryMehta]

@vmarchman I was about to type that your numbers make no sense and then I realised they are weeks and months and the premature is currently treated the same as the age.

I'm not sure how clinical reports could be ok because the clinical reports were calculating the adjusted age incorrectly. I was adding/subtracting the number of weeks (rather than months). I have corrected for CAT, WS and WG.

I do not think you're right on point 2. If you want benchmarks for the adjusted age, then you want to use the benchmark cohort for that age so if we have a benchmark for that age group we should use it. If you want benchmarks for the birth age then you select the other csv report and get benchmarks for the chronological age. This is what my test reports are showing. If you're getting something different could you let me know the study so I can look at the results you're getting.

I am loading the adjusted age correction for all clinical reports now

[vmarchman]

Hi @Henry @.***> Ok, I see what you mean. But, let's take this one test case that I've been using.

For a WS child, age = 34 and the child is 8 weeks (2 months) premature.

For the regular Clinical Report and csv, the child's age = 34 and the child's benchmark age is 30 months. CORRECT. We are not correcting for prematurity in this case.

But, for reports with adjusted benchmarks:

For the clinical report, the age (34) and benchmark age (30) are Correct. But, the adjusted age is reported as 36, i.e., 2 months older than the Chronological age. It should be 32, i.e., 2 months younger, 34 months - 8 weeks (2 months) premature.

For the CSV, the adjusted age is reported as 28 (different than in the clinical report). This is two months younger than the Benchmark age, rather than 2 months younger than the chronological age.

---

I also tested the CAT with a similar aged and premature child. The Adjusted age in the CR is 2 months OLDER (rather than younger) than the chron age. The adjusted age in the csv is the same as the chron, so not adjusted at all.

-vm

On Tue, Feb 27, 2024 at 9:40 AM Henry Mehta @.***> wrote:

@vmarchman https://github.com/vmarchman I was about to type that your numbers make no sense and then I realised they are weeks and months and the premature is currently treated the same as the age.

I'm not sure how clinical reports could be ok because the clinical reports were calculating the adjusted age incorrectly. I was adding/subtracting the number of weeks (rather than months). I have corrected for CAT, WS and WG.

I do not think you're right on point 2. If you want benchmarks for the adjusted age, then you want to use the benchmark cohort for that age so if we have a benchmark for that age group we should use it. If you want benchmarks for the birth age then you select the other csv report and get benchmarks for the chronological age. This is what my test reports are showing. If you're getting something different could you let me know the study so I can look at the results you're getting.

I am loading the adjusted age correction for all clinical reports now

— Reply to this email directly, view it on GitHub https://github.com/langcog/web-cdi/issues/507#issuecomment-1967275468, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB2TUTCPY6YPZXHEPMECJRTYVYLALAVCNFSM6AAAAABDRR5APSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRXGI3TKNBWHA . You are receiving this because you were mentioned.Message ID: @.***>

[HenryMehta]

@vmarchman

For not CAT administrations It looks like we've been calculating Adjusted Age incorrectly for all existing administrations. The issue only occurs if the Benchmark Cohort Age is lower than the administration age, in which case the Adjusted Age is adjusted from the Benchmark Cohort Age rather than the administration age. I have corrected this issue and reapplied in test to your administrations. Please check and confirm. I am concerned the recalc ran too quickly so if it hasn't worked, please create a new administration with the same test you had above and see if that works.

If this has worked I'll have to update the Summary data for all existing administrations in production.

For CAT administrations I have corrected the way the Adjusted Age is calculated. It should now be correct

[vmarchman]

Hi @Henry @.***> I did some more testing and we're getting closer, but still not there.

ok - for a CAT administration. I have a 23 monther who is 8 weeks premature. In the csv, his adjusted age is still 23 and so is his benchmark age - both should be 21. In the clinical report for that child, the adjusted age is correct (21), but the benchmark age is still 23.

It was a new administration.

For a non-CAT (WS kid) in the same sort of situation, it is correct in both the csv and in the clinical report.

I've put screen shots in slack.

The other thing that seems to be happening (somewhat randomly, unfortunately), is that I enter in the demographics 8 weeks early, and in the csv, sometimes, it reverts to 7 weeks...?

-vm

On Wed, Feb 28, 2024 at 12:18 AM Henry Mehta @.***> wrote:

@vmarchman https://github.com/vmarchman

For not CAT administrations It looks like we've been calculating Adjusted Age incorrectly for all existing administrations. The issue only occurs if the Benchmark Cohort Age is lower than the administration age, in which case the Adjusted Age is adjusted from the Benchmark Cohort Age rather than the administration age. I have corrected this issue and reapplied in test to your administrations. Please check and confirm. I am concerned the recalc ran too quickly so if it hasn't worked, please create a new administration with the same test you had above and see if that works.

If this has worked I'll have to update the Summary data for all existing administrations in production.

For CAT administrations I have corrected the way the Adjusted Age is calculated. It should now be correct

— Reply to this email directly, view it on GitHub https://github.com/langcog/web-cdi/issues/507#issuecomment-1968449840, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB2TUTFWVOA2VUXITZNVGLLYV3R37AVCNFSM6AAAAABDRR5APSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRYGQ2DSOBUGA . You are receiving this because you were mentioned.Message ID: @.***>

[HenryMehta]

@vmarchman I am travelling tomorrow so will try and look at this either tonight, Friday or over the weekend. I think the 7/8 weeks premature is likely to be the mouse moving or you hitting the down arrow by mistake and causing the number to scroll

[vmarchman]

@Henry @.***> ok - safe travels!

On Wed, Feb 28, 2024 at 9:45 AM Henry Mehta @.***> wrote:

@vmarchman https://github.com/vmarchman I am travelling tomorrow so will try and look at this either tonight, Friday or over the weekend. I think the 7/8 weeks premature is likely to be the mouse moving or you hitting the down arrow by mistake and causing the number to scroll

— Reply to this email directly, view it on GitHub https://github.com/langcog/web-cdi/issues/507#issuecomment-1969518308, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB2TUTAAZFBQYFSSIFNMDO3YV5ULTAVCNFSM6AAAAABDRR5APSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRZGUYTQMZQHA . You are receiving this because you were mentioned.Message ID: @.***>

[HenryMehta]

@vmarchman Sorry for the week's delay in doing this but I have now made the amendments, tested locally and deployed

[vmarchman]

No worries @Henry @.***>

I've tested the CAT and they seem good.

We still have issues with the WS and WG though.

On the WS, I have a participant who is 25 months with an adjusted age of 24 months. The benchmarking age should be 24 months. It is correct on the clinical report, but not the csv.

On the WG, the clinical reports are correct, but in the adjusted benchmarking csvs, the benchmarking age is not the adjusted age, but rather the chron age. Adjusted age is correct, but the benchmarking age is not.

Let me know if you want to see examples!

-vm

On Wed, Mar 6, 2024 at 10:46 PM Henry Mehta @.***> wrote:

@vmarchman https://github.com/vmarchman Sorry for the week's delay in doing this but I have now made the amendments, tested locally and deployed

— Reply to this email directly, view it on GitHub https://github.com/langcog/web-cdi/issues/507#issuecomment-1982593859, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB2TUTHVLDICVHDAJ5VS4H3YXAELPAVCNFSM6AAAAABDRR5APSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBSGU4TGOBVHE . You are receiving this because you were mentioned.Message ID: @.***>

[HenryMehta]

@vmarchman I have deployed a fix. It needs the summary scores to update to work and I've set yours to update in a couple of minutes but the might take 30 to complete. If they don't you may need to create the criteria for the test again

[vmarchman]

@Henry @.***>

I think it's right!

for English WG, WS, and CAT.

I suddenly think I need to test for Spanish too? and all the other forms???

On Thu, Mar 7, 2024 at 12:14 PM Henry Mehta @.***> wrote:

@vmarchman https://github.com/vmarchman I have deployed a fix. It needs the summary scores to update to work and I've set yours to update in a couple of minutes but the might take 30 to complete. If they don't you may need to create the criteria for the test again

— Reply to this email directly, view it on GitHub https://github.com/langcog/web-cdi/issues/507#issuecomment-1984347949, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB2TUTHX6A2PSWNUHF7WCRDYXDDDNAVCNFSM6AAAAABDRR5APSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBUGM2DOOJUHE . You are receiving this because you were mentioned.Message ID: @.***>

[HenryMehta]

@vmarchman The calculations are the same for the other forms, certainly for WS and WG. It would be worth testing Spanish CAT since it does use a different html file.

[vmarchman]

Hi @Henry @.***> Mexican CAT looks good!

The only thing is on the clinical reports, it should have simply "Jackson-Maldonado et al (2003)" on the bottom - not 3rd Edition.

We are good to go!!

(Thanks - I know this was a pain.)

On Thu, Mar 7, 2024 at 11:26 PM Henry Mehta @.***> wrote:

@vmarchman https://github.com/vmarchman The calculations are the same for the other forms, certainly for WS and WG. It would be worth testing Spanish CAT since it does use a different html file.

— Reply to this email directly, view it on GitHub https://github.com/langcog/web-cdi/issues/507#issuecomment-1985181923, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB2TUTAJ4HPYKVJIVJ6CNGDYXFRZZAVCNFSM6AAAAABDRR5APSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBVGE4DCOJSGM . You are receiving this because you were mentioned.Message ID: @.***>

[HenryMehta]

@vmarchman Updates now completed. Everything should work in production

[vmarchman]

Thanks @Henry @.***> !

On Sun, Mar 10, 2024 at 1:32 PM Henry Mehta @.***> wrote:

@vmarchman https://github.com/vmarchman Updates now completed. Everything should work in production

— Reply to this email directly, view it on GitHub https://github.com/langcog/web-cdi/issues/507#issuecomment-1987353737, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB2TUTFPXSTEMLF4RQWXXFDYXS7M7AVCNFSM6AAAAABDRR5APSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBXGM2TGNZTG4 . You are receiving this because you were mentioned.Message ID: @.***>