Re-ensure that all omb_idea canonical URLs are in the index

GSA / site-scanning

The central repository for the Site Scanning program

https://digital.gov/site-scanning

11 stars 2 forks source link

Re-ensure that all omb_idea canonical URLs are in the index #1030

Open gbinal opened 2 weeks ago

gbinal commented 2 weeks ago

Following up from here (and an earlier version of this here)....

There are 249 sites in the omb_idea file that are missing from the index.

34 are .govs, but I've checked and all of them shouldn't have been included in the index
1 is a .mil and I've checked and it shouldn't have been included.
119 end in .com
60 end in .org
The remaining 36 all have other TLDs
(34+1+119+60+36=250 - close enough)

gbinal commented 2 weeks ago

I can try to add back in the gov and mil ones and, except for those that are out of date (e.g. the domain is no longer live) or isn't federal, they'll be added in.

gbinal commented 2 weeks ago

done

gbinal commented 2 weeks ago

reopening b/c the new ignore filters I'm putting in will impact this and I'll need to do this again. ;/