ctti-clinicaltrials / aact

Improving Public Access to Aggregate Content of ClinicalTrials.gov
http://aact.ctti-clinicaltrials.org
MIT License
123 stars 33 forks source link

studies.limitations_and_caveats missing #925

Open AstrorEnales opened 2 years ago

AstrorEnales commented 2 years ago

Recently i noticed the field limitations_and_caveats of the studies table not being filled any more.

In the pipe-delimited-export from september 2021 there is: NCT02799667 -> "Early termination secondary to slow recruitment."

Which is still available on clinicaltrials.gov online: https://clinicaltrials.gov/ct2/show/results/NCT02799667?view=results#limit

However, in the latest downloads and also cloud database the fields are always empty: SELECT nct_id, limitations_and_caveats FROM ctgov.studies WHERE nct_id='NCT02799667';

micronix commented 2 years ago

Thank you for alerting us. The location of this field in the api response might have changed when we started using the new api. We will work on a fix for this.

AstrorEnales commented 2 years ago

I further investigated and the field "LimitationsAndCaveats" is defined in the study structure under "MoreInfoModule", but now named "LimitationsAndCaveatsDescription".

<Struct Name="MoreInfoModule">
    <Field Name="LimitationsAndCaveatsDescription" DEDLink="https://prsinfo.clinicaltrials.gov/results_definitions.html#Result_LimitationsAndCaveats_description"/>
    [...]
</Struct>

In your code it seems to be accessed under the old name: ['FullStudiesResponse']['FullStudies'].first['Study']['ResultsSection']['MoreInfoModule']['LimitationsAndCaveats']

Nontheless, the field is still not provided by the full_studies endpoint of ClinicalTrials! https://clinicaltrials.gov/api/query/full_studies?expr=AREA%5BNCTId%5DNCT02799667&min_rnk=1&max_rnk=&fmt=json

If requested directly using the study_fields endpoint, the correct text is returned: https://clinicaltrials.gov/api/query/study_fields?expr=AREA%5BNCTId%5DNCT02799667&fields=NCTId,LimitationsAndCaveatsDescription&min_rnk=1&max_rnk=&fmt=json

Either ClinicalTrials.gov has an issue in their api themselves, or the field needs to be requested separately, which wouldn't be great.

micronix commented 2 years ago

Yes, it looks like that is the case, we will update to the new name, will have it running the new code tomorrow

micronix commented 2 years ago

Hello,

I misread what you found and yes it looks like this is not a simple fix, but clinicaltrials.gov might need to help us because this is not being returned in the full studies api endpoint. We will contact them and let you know

micronix commented 2 years ago

@AstrorEnales we spoke with the team at clinicaltrials.gov and we are waiting to hear back what the issue might be, we will keep you updated.