briri
commented
6 days ago We need to update the index format for DMPs so that we are handling most of the normalization tasks on our side before sending to COKI. See this writeup for what is currently being done:
title and abstract
- removing HTML
- Stop converting everything to lowercase
- remove all quotation marks
Project dates
- Make sure start and end are actually dates
- set the end date to (start date + 3 years) if it is less than 3 years
Affiliation/Funders
- Remove parentheses and their contents.
- Trim and lowercase each name.
- Convert empty strings to null.
- Remove ROR domain (just keep the id suffix)
- Trim and lowercase each opportunity number
Grant Ids
- Extract NSF Award IDs from NSF Award URLs
- Extract NSF ID from NSF publication URLs
- Extract any number with 5 or more digits.
People
- Replace instances of one or more spaces with a single space.
- Remove parentheses and their contents.
- Re-order names as “first last”.
- Trim and lowercase
- Convert empty string to null
- Remove ORCID domain (just keep id suffix)
Once these changes have been made to the indexer Lambda, rebuild the index.
Build out a Lambda function based on the the script we generated as part of #15
This will need to be run every week on Saturday morning. The file will need to be written to an S3 bucket defined in aws repo issue #175.