It would be helpful to document some useful searches one could make in SQLite with the output of a crawl.
For example, "find all pages that link to /about-us/", and save as CSV:
sqlite> .mode csv
sqlite> .output about_us_links.csv
sqlite> SELECT DISTINCT url, json_each.value link FROM cfpb, json_each(json(cfpb.contentLinks)) WHERE json_each.value LIKE '/about-us/%' ORDER BY url, link;
sqlite> .output stdout
It would be helpful to document some useful searches one could make in SQLite with the output of a crawl.
For example, "find all pages that link to /about-us/", and save as CSV: