Add National Science Foundation

[spulec]

Currently getting a few 404s:

http://www.nsf.gov/oig/search/m00090035.pdf http://www.nsf.gov/oig/search/m01010002.pdf http://www.nsf.gov/oig/search/m01020006.pdf http://www.nsf.gov/oig/search/m97100037.pdf

If you change the m to M, they resolve. I'm going to reach out to them to see if they can fix the links. If they don't respond, I can add a fix to this.

I'm also going to see if I can get them to add a published date for the one report in REPORT_PUBLISHED_MAP.

[konklone]

Hmm, re: your note:

# Notes for IG's web team:
# - https://www.nsf.gov/oig/search/ encounters an error when using https.
# while it works if just using http.

I get the same application error for both URLs:

• http://www.nsf.gov/oig/search/results.cfm
• https://www.nsf.gov/oig/search/results.cfm

Unless it only happens with a POST? Not totally sure how that URL works. Is it possible that it's not related to the protocol, but to your current HTTP session? Each protocol would use a different session/cookies, so is it possible that you had valid session data for one protocol, but not the other?

[konklone]

Anyway, not a blocker, since it works and all! Thanks for doing this, @spulec.

[spulec]

I think it might just be the POST, but I'm not positive.

CASE_REPORTS_DATA = {
  'sortby': 'rpt_num',
  'sballfrm': 'Search',
}
import requests

http_response = requests.post(
  'http://www.nsf.gov/oig/search/results.cfm',
  data=CASE_REPORTS_DATA,
)

https_response = requests.post(
  'https://www.nsf.gov/oig/search/results.cfm',
  data=CASE_REPORTS_DATA,
)

Both return status 200, but https_response.content shows an error while http_response.content does not. It's possible that adding some additional headers would get the https one to work.

[konklone]

Thanks for investigating further -- doesn't seem worth more work, anyway, it works.