uspto.gov disappeared

[tdlowden]

Just got an email from folks at uspto.gov wondering why it was no longer on pulse.cio.gov. Not sure why this happened - still appears on the DAP second-level domains list, and not on the ineligible.yaml file to remove certain domains.

[gbinal]

Thanks, @tdlowden. I've confirmed that it's happening and agree with your initial analysis. I'm not sure what's up but will try to figure it out asap. I'll follow up here with updates.

[tdlowden]

hey folks. any update on this?

[konklone]

Our scanner consistently can't connect to uspto.gov on the last several scans:

• March 10th: https://s3-us-gov-west-1.amazonaws.com/cg-4adefb86-dadb-4ecf-be3e-f1c7b4f6d084/archive/2017-03-10/cache/pshtt/uspto.gov.json
• March 1st: https://s3-us-gov-west-1.amazonaws.com/cg-4adefb86-dadb-4ecf-be3e-f1c7b4f6d084/archive/2017-03-01/cache/pshtt/uspto.gov.json
• Feb 20th: https://s3-us-gov-west-1.amazonaws.com/cg-4adefb86-dadb-4ecf-be3e-f1c7b4f6d084/archive/2017-02-20/cache/pshtt/uspto.gov.json

However, it was able to if we go back to Jan 30th:

• https://s3.amazonaws.com/pulse.cio.gov/archive/2017-01-30/cache/pshtt/uspto.gov.json

We did change servers between those two time frames (which is why the archive URL is different).

When I check from our current scanning server, we can't talk to uspto.gov at all -- even just running curl --head http://uspto.gov hangs. From my local laptop, I can access it. I get the same DNS lookup results on both our server and my laptop.

So offhand, I'm wondering if maybe USPTO blocked our scanning server. If so, this would actually be the first time I've found that to have happened in Pulse's 2 years of scanning.

[konklone]

Alternatively, USPTO could have whitelisted our previous server, and then we switched servers our IP changed. I don't recall a conversation with them about it, but there didn't necessarily need to be one.

Our IP address is different, and is no longer static (since we don't have a website hosted on the same box anymore, it's just back-end stuff now). So our IP isn't guaranteed to remain the same over time. If they are white- or blacklisting our scans, I would request they do so by User-Agent and not by IP address. We use a User-Agent of github.com/18f/domain-scan, pshtt.py, in part to put sysadmins at ease.

[tdlowden]

Thanks for this description, @konklone. I'm circling back with the USPTO team to ask.

[gbinal]

FYI, it appears that USPTO resolved this, so I'm closing the issue.