This repository contains Mozilla Data Team's:
For more information, see https://mozilla.github.io/bigquery-etl/
gcloud auth login --update-adc
to authenticate against GCP.shared-prod
if making changes to production systems. Run gcloud auth login --update-adc --project=moz-fx-data-shared-prod
(if you have not run it previously).Clone the repository
git clone git@github.com:mozilla/bigquery-etl.git
cd bigquery-etl
Install the bqetl
command line tool
./bqetl bootstrap
Install standard pre-commit hooks
venv/bin/pre-commit install
Finally, if you are using Visual Studio Code, you may also wish to use our recommended defaults:
cp .vscode/settings.json.default .vscode/settings.json
cp .vscode/launch.json.default .vscode/launch.json
And you should now be set up to start working in the repo! The easiest way to do this is for many tasks is to use bqetl
. You may also want to read up on common workflows.
bqetl
To push a new version of bqetl
to PyPI, update the version
in pyproject.toml
. The version numbers follow the CalVer scheme, with the Micro version numbers starting at 1. For example, for the first package version getting published in March 2024, the version would be 2024.3.1
.