Create dataset+table (if not exists) and upload data. The table should be specified in the fully qualified format.
python -m scraper upload --table_id project.dataset.table --data scraped.csv
If you need to specify a Big Query service account file, do so using the '--key-path' argument:
python -m scraper upload --key-file service_account.json --table-id ... --data ...
See Issue #11 for context. This PR introduces functions for interacting with the BigQuery database. It also restructures the code to run as a module.
Install dependencies before running this tool:
pip install -r requirements.txtScrape the data:
python -m scraper scrape --output csv > scraped.csvCreate dataset+table (if not exists) and upload data. The table should be specified in the fully qualified format.
python -m scraper upload --table_id project.dataset.table --data scraped.csvIf you need to specify a Big Query service account file, do so using the '--key-path' argument:
python -m scraper upload --key-file service_account.json --table-id ... --data ...