consider ELT with BigQuery as pseudo-schemaless db

[maxheld83]

These approaches might work for us. Have to do more research, just to remember the links:

• https://articles.anelen.co/elt-google-cloud-storage-bigquery/
• https://cloud.google.com/blog/products/gcp/bigquery-lazy-data-loading-sql-data-languages-ddl-and-dml-partitions-and-half-a-trillion-wikipedia-pageviews
• https://medium.com/google-cloud/bigquery-lazy-data-loading-ddl-dml-partitions-and-half-a-trillion-wikipedia-pageviews-cd3eacd657b6
• https://cloud.google.com/bigquery/docs/hive-partitioned-loads-gcs

[maxheld83]

this is also what (some) big nerds apparently do: https://github.com/mozilla/gcp-ingestion/issues/423

[maxheld83]

this is a good summary of the case for ELT https://www.stitchdata.com/resources/what-is-elt/

[maxheld83]

prep been implemented here:

• https://github.com/The-Academic-Observatory/observatory-platform/blob/ecf20bae0d5bada3e44cd1c7c236958ff0a77364/observatory-dags/observatory/dags/telescopes/crossref_metadata.py#L124

and schema here: https://github.com/The-Academic-Observatory/observatory-platform/blob/6f4a67b6119492d4ebc55e89be1c4a73c0be833c/observatory-dags/observatory/dags/database/schema/crossref_metadata_2020-09-01.json