cc-archive / cccatalog

[PROJECT TRANSFERRED] Mapping the commons towards an open ledger and cc search.
https://github.com/WordPress/openverse-catalog
MIT License
63 stars 60 forks source link

[Bug] Add variables to env.template #406

Closed mathemancer closed 4 years ago

mathemancer commented 4 years ago

Bug Description

Some environment variables are missing from

src/cc_catalog_airflow/env.template

This means that the dev environment setup instructions from README.md are currently a bit broken.

(The work, technically, but the resulting setup is suboptimal for testing).

In particular, we should add:

DATA_GOV_API_KEY=not_set

...

LOADER_FILE_AGE=1

To Reproduce

  1. Follow the README.md to setup the local development environment.
  2. Turn on the smithsonian_workflow, flickr_workflow, and tsv_to_postgres_loader
  3. Watch the smithsonian_workflow fail, and watch the tsv_to_postgres_loader take at least 15 minutes after the completion of flickr_workflow to actually load anything.

Expected behavior

Everything should work properly, and the time before tsv_to_postgres_loader loads a file should be set more conveniently for testing.