outbreak-info / outbreak.info-resources

A curated repository of metadata of resources on COVID-19 and SARS-CoV-2
MIT License
0 stars 4 forks source link

data missing from ES index #181

Closed gtsueng closed 2 years ago

gtsueng commented 2 years ago

This branch of the Litcovid parser successfully adds the correction field to a relevant litcovid docs when run on dev; however, the corresponding docs containing this property disappear altogether from the dev api. The mapping file referenced by the upload.py file for this branch is the v3 map found here.

The issue is suspected to be due to ES mapping/indexing since the docs are found in the parsed dump, but are not accessible via the dev api.

An example doc: pmid33563391 appears in https://api.outbreak.info/resources/resource/pmid33563391 but gives a 404 error when the same call is posted to the dev api.

The doc contains the desired information: {'@context': {'schema': 'http://schema.org/', 'outbreak': 'https://discovery.biothings.io/view/outbreak/'}, '@type': 'Publication', 'author': [{'@type': 'outbreak:Person', 'affiliation': [{'@type': 'outbreak:Organization', 'name': 'Doncaster and Bassetlaw Teaching Hospitals, Doncaster, UK, Honorary Lecturer at Sheffield University, Sheffield, UK.'}], 'familyName': 'Hussain', 'givenName': 'Abdulzahra', 'name': 'Abdulzahra Hussain'}, {'@type': 'outbreak:Person', 'affiliation': [{'@type': 'outbreak:Organization', 'name': 'Bariatric Unit, Department of General Surgery, Sunderland Royal Hospital, Sunderland, UK.'}], 'familyName': 'Mahawar', 'givenName': 'Kamal', 'name': 'Kamal Mahawar'}, {'@type': 'outbreak:Person', 'affiliation': [{'@type': 'outbreak:Organization', 'name': 'Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, 1277 Jiefang Avenue, Wuhan, Hubei Province, China.'}], 'familyName': 'Xia', 'givenName': 'Zefeng', 'name': 'Zefeng Xia'}, {'@type': 'outbreak:Person', 'affiliation': [{'@type': 'outbreak:Organization', 'name': 'The First Affiliated Hospital of Jinan University, 613 Huangpu Avenue West, Guangzhou, Guangdong Province, China.'}], 'familyName': 'Yang', 'givenName': 'Wah', 'name': 'Wah Yang'}, {'@type': 'outbreak:Person', 'affiliation': [{'@type': 'outbreak:Organization', 'name': "Bariatric Unit, Princess Royal University Hospital, King's College Hospitals NHS Foundation Trust, London, UK."}], 'familyName': 'El-Hasani', 'givenName': 'Shamsi', 'name': 'Shamsi El-Hasani'}], 'publicationType': ['Published Erratum', 'Retraction of Publication'], '_id': 'pmid33563391', 'curatedBy': {'@type': 'schema:WebSite', 'name': 'litcovid', 'curationDate': '2021-09-09', 'url': 'https://www.ncbi.nlm.nih.gov/research/coronavirus/publication/33563391'}, 'name': 'Retraction notice to Obesity a nd Mortality of COVID-19.Meta-analysis [Obesity Research & Clinical Practice 14/4 (2020) 295-300].', 'identifier': '33563391', 'pmid': '33563391', 'journalName': 'Obesity research & clinical practice', 'volumeNumber': '15', 'journalAbbreviation': 'Obes Res Clin Pract', 'issueNumber': '1871-403X', 'doi': '10.1016/j.orcp.2020.12.008', 'url': 'https://www.doi.org/10.1016/j.orcp.2020.12.008', 'correction': [{'@type': 'Correction', 'identifier': 'pmid32660813', 'url': 'https://www.ncbi.nlm.nih.gov/research/coronavirus/publication/32660813', 'pmid': '32660813', 'correctionType': 'retraction of'}], 'dateModified': '2021-03-12', 'dateCreated': '2021-02-11'}

gtsueng commented 2 years ago

Somehow it's working in the production API.