Load analytics from BigQuery

[MrOrz]

This PR refactors fetchStatsFromGA script to read from BigQuery instead of deprecating Universal Analytics.

Design doc: https://g0v.hackmd.io/cJFFXVzgT4OFT6bBLtulGg#-API

• It uses 3 separate SQL commands to read web, LIFF and LINE traffic respectively from BigQuery tables, and in the implementation we join the 3 SQL command result to form the table in analytics index's form
  • Original logic was relatively complex because it needs to read data out of Universal Analytics. GA4 events are much straightforward; and BigQuery SQL really simplifies things a lot.
• In NodeJS script we just map individual SQL result row to analytics documents.
  • Use dataloader to load docUserId and docAppId fields
• Integration test that puts test fixtures in BigQuery and test if SQL reads stuff and aggrergates into analytics as expected

[coveralls]

coverage: 88.133% (-0.2%) from 88.352% when pulling ea10b3c06da0b0c9fcfb98e29ca57eb26b2288d0 on johnson/bq-analytics into 7c740319442931a04b4add1dc6ba91a6d72b0387 on master.

[MrOrz]

Deployment: set these env vars

      - TIMEZONE=Asia/Taipei
      - LINE_BOT_EVENT_DATASET_ID=
      - GA4_DATASET_ID=
      - GA_WEB_STREAM_ID=
      - GA_LIFF_STREAM_ID=
      - GOOGLE_APPLICATION_CREDENTIALS=/data/...

Cronjob that updates "yesterday"'s stat (on staging)

15 16 * * * cd answerfamily-deploy; TZ='Asia/Taipei' YESTERDAY=`date --date='yesterday' +'%Y%m%d'` ./compose-all exec -T cofacts-api-staging node build/scripts/fetchStatsFromGA.js --startDate=$YESTERDAY --endDate=$YESTERDAY >> cron.log 2>&1

After updating from 6/5 to 6/26 + 6/27 (today)