revise crawl script to create index and add to git

[rlskoeser]

implemented in https://github.com/Princeton-CDH/cdh-ansible/pull/105

confirmed that the most recent crawl includes base sitemap, 404, and 500 pages

browsertrix does not seem to parse and crawl sitemaps