Sitemap scraper

[nfmcclure]

Added files, readme, and sample URL list for sitemap scraper.

[josh-chamberlain]

Thanks for the submission @nfmcclure! I'm testing this now but stuck:

(venv) $ scrapy crawl sitemapspider
Scrapy 2.7.1 - no active project

Unknown command: crawl

Anything else I should do after setting up the venv?

[nfmcclure]

Yup sorry- two changes needed,

1. Added scrapy to requirements.txt.
2. change directory to sitemap before running scrapy crawl sitemapspider.

Let me know if that helps.

Edit- ok re-tested in a completely new venv. Fixed some requirements.

[josh-chamberlain]

This is weird, but still not working! I can try more intensively later 🤷

[nfmcclure]

That is strange! I haven't run into that before. But it seems that can happen, as tackled here:

https://stackoverflow.com/questions/45345377/python-module-not-found-even-though-requirement-already-satisfied-in-pip

[josh-chamberlain]

@nfmcclure cool—from the venv, uninstalling and reinstalling xmltodict (which wasn't installed outside the venv) worked. I was able to get this to run! I'll wait for @thejqs to approve but LGTM

[josh-chamberlain]

Related to https://github.com/Police-Data-Accessibility-Project/data-source-identification/issues/11

[josh-chamberlain]

I used this for some Southeast Arkansas agencies with mixed success:

attempt 1: sample_host_sites.txt 20230131_122648_output.csv

attempt 2: sample_host_sites.txt 20230131_123505_output.csv