data/redirects.json contains the link between ARKs and OSTI DOI. The problem is that when we migrate dataset over, Scraper uses the URL from the DOI link, which now redirects. As such, we aren't able to get
the ARK. Unsure how we will fix this.
This caused us to find more records that normal. I investigated and found many not to be new and already have an ARK.
data/redirects.json contains the link between ARKs and OSTI DOI. The problem is that when we migrate dataset over,
Scraperuses the URL from the DOI link, which now redirects. As such, we aren't able to get the ARK. Unsure how we will fix this.This caused us to find more records that normal. I investigated and found many not to be new and already have an ARK.