Check via CDX API if URL archived within the last 30 days

edgi-govdata-archiving / eot-nomination-tool

📚 Chrome extension to nominate government data that needs to be preserved

https://chrome.google.com/webstore/detail/nominationtool/abjpihafglmijnkkoppbookfkkanklok

GNU General Public License v3.0

20 stars 10 forks source link

Check via CDX API if URL archived within the last 30 days #49

Closed atesgoral closed 7 years ago

atesgoral commented 7 years ago

Changes to the URL field (keyup, paste, change events) trigger the CDX URL lookup with a debounce of 500ms (i.e. 500ms after user's last action)
We only fetch the timestamp field to reduce payload returned from CDX
CDX results are normalized into an array of objects with the timestamp properties converted to JS Dates
Only one result is requested right now (the most recent)
No visual indication of this background check is given unless the URL is found to be archived within the last 30 days -- in which case a warning is shown
Consolidated success/error notifications to use Bootstrap's alerts
Did some miscellaneous refactoring

atesgoral commented 7 years ago

@titaniumbones The feature is already in this PR. I broke my own principle of keeping PRs topical and did the new feature + refactoring in one shot.

titaniumbones commented 7 years ago

@atesgoral did you see @mhucka 's notes about the 503? Does it give you any concerns? I guess now that you're not getting the 503's it's hard to check the Retry-After header.

atesgoral commented 7 years ago

Yeah, it's too late now to check if there was a Retry-After header.

Wishfully thinking that it was just a fluke :)

It would be interesting to hear from the IA folks what they think about the 503 responses.

I'm looking around for a cloud-based API caching gateway. Amazon's API Gateway can be configured to cache API responses from an endpoint for a certain amount of time, but it's not available on the free tier. Also, it's optimistic to think that there will be a lot of cache hits to be able to take advance of an API response cache.

atesgoral commented 7 years ago

Realized I forgot to check the URL when the popup initially loads :) Adding that now.

mhucka commented 7 years ago

Just a quick note to say I haven't heard anything back yet. Will keep you posted. I'm sure the IA people are beyond busy these days ...

atesgoral commented 7 years ago

@mhucka, @titaniumbones It could be best if I just leave the feature code in, but just disable the CDX lookup so that we can at least merge this PR with all the refactoring that was done. Then it would be just flipping a switch to re-enable the lookup.

mhucka commented 7 years ago

@atesgoral Sounds like a most excellent plan!

atesgoral commented 7 years ago

URL checking disabled.

titaniumbones commented 7 years ago

Excellent, merging. @mhucka, if somehow you hear back form Jefferson that the problem is unrelated to server load, we can revert 18e69ee and take this live at any point.

titaniumbones commented 7 years ago

Should i delete the branch or keep it open?

atesgoral commented 7 years ago

Safe to delete! Deleted.