Updating the database of records

[Frijol]

Our search indexing at present comes from a snapshot of the EPA EIS Database

If we were to update this programmatically, it would look something like:

• [ ] Create a programmatic way to index what EIS records exist (blank search on the EPA EIS site returns all, can get a CSV of much info not including IDs)
• [ ] Create a programmatic way to map IDs onto the data the EPA's EIS site gives us (this lets us construct links to the EPA EIS pages for those records)
• [ ] OCR indexing for bleve?
• [ ] Design a chron job to regularly (how often?) run the process as a whole

This is something that should happen eventually in the course of developing this tool. But I have two questions to determine priority:

1) How often does this need to happen? (I'll try to get an answer to this via user research e.g. "how often do you look for EIS records in this dataset? How often do you use a record from the last year?") 2) How much of a pain is it to index a new snapshot using our current process? (@b5 can you or someone else familiar with the process provide an outline of the current process?)

[Frijol]

Latest from Jessica:

• The database is not useful for her current work without regular updating– she needs to access EIS's that are still in draft
• EIS's in draft are posted to the database we draw from (so, wouldn't need to poll a separate database) (she also notes that there is a filter in that database for draft EIS's)
• She suggests weekly is an appropriate update schedule

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in seven days if no further activity occurs. If it should not be closed, please comment! Thank you for your contributions.