cfpb / cfgov-crawler-app

An electron app which crawls consumerfinance.gov and gathers interesting data
6 stars 8 forks source link

Add documentation for querying SQLite database #7

Open chosak opened 2 years ago

chosak commented 2 years ago

It would be helpful to document some useful searches one could make in SQLite with the output of a crawl.

For example, "find all pages that link to /about-us/", and save as CSV:

sqlite> .mode csv
sqlite> .output about_us_links.csv
sqlite> SELECT DISTINCT url, json_each.value link FROM cfpb, json_each(json(cfpb.contentLinks)) WHERE json_each.value LIKE '/about-us/%' ORDER BY url, link;
sqlite> .output stdout