Analytics for LIFF views

[MrOrz]

Fixes #281

• analyticsLoader: Let analyticsLoader return the entire Analytics object, so that its result can feed in the new Analytics type
• New API ListAnalytics
  • Lists analytics of any article or reply just by their ID, without the need to query the article first
  • Will be helpful as we separate the data loading of analytics from article
  • Can also search for docUserId so that we can list out analytics for a specific user's contribution
  • Can list liff field
• Cron job fetchStatsFromGA populates liff field
  • liff field's structure (array) is different from other fields such as webUser, so its processing logic is separate from other fields
  • liff field's source data requires a breakdown dimension (by ga:source), so existing data loading logic for other fields such as webUser also cannot apply
  • That's why LIFF loading logic updateLiffStats() is entirely separate from other fields' logic updateStats()
• Refactor: in media-manager integration test, make the wait for thumbnail more meaningful and reduce flaky test

Validation

LIFF data filled byfetchStatsFromGA is consistent with Google data studio's view count.

[MrOrz]

Analytics 歷史資料拉取與 Analytics reporting API quota 估算

總資料量

2021/9/7 啟用 LIFF 至 2022/7/27： 316,337 view count for all replies 假設每個 analytics record (per day doc ID) 都只有一個 page view、並考慮 script 會拉 article 與 reply 的 page view，316k view會有 316k 2 = 630k 的 analytics records要拉。

API call 與 quota 使用量計算

由於我們每次 API call 會拉 10000 筆下來，630k 只要拉 63 次。

Google analytics quota & limits

• 50,000 requests per day
  • 10,000 per view (profile) per day
  • 10 concurrent requests per view (profile)
• 2,000 requests per 100 seconds
• 100 requests per 100 seconds per IP
• 10 requests per second per user (IP & user can be forwarded via userIP or quotaUser parameter)

應該都沒超過。

執行時間估算

跑 8 天資料約 60 seconds、10 request

node_modules/.bin/babel-node src/scripts/fetchStatsFromGA.js --loadScript --startDate=2022-07-18
fetched 10000 starting at 0 out of total 42764 rows for WEB article with next pageToken 10000
fetched 4975 starting at 0 out of total 4975 rows for WEB reply
no more rows to fetch for WEB reply
fetched 10000 starting at 10000 out of total 42767 rows for WEB article with next pageToken 20000
no more rows to fetch for WEB reply
fetched 10000 starting at 20000 out of total 42767 rows for WEB article with next pageToken 30000
no more rows to fetch for WEB reply
fetched 10000 starting at 30000 out of total 42767 rows for WEB article with next pageToken 40000
no more rows to fetch for WEB reply
fetched 2767 starting at 40000 out of total 42767 rows for WEB article
fetched 3156 starting at 0 out of total 3156 rows for LINE article
fetched 2191 starting at 0 out of total 2191 rows for LINE reply
fetched 2653 starting at 0 out of total 2653 rows for LIFF article
fetched 3063 starting at 0 out of total 3063 rows for LIFF reply

去年 9 月至今約 330 天 --> 預計 40 分鐘、 420 個 request（大高估）

實測

404s，53 API calls

$ node_modules/.bin/babel-node src/scripts/fetchStatsFromGA.js --loadScript --startDate=2021-09-01

fetched 10000 starting at 0 out of total 274617 rows for WEB article with next pageToken 10000
fetched 10000 starting at 0 out of total 26291 rows for WEB reply with next pageToken 10000
fetched 10000 starting at 10000 out of total 274617 rows for WEB article with next pageToken 20000
fetched 10000 starting at 10000 out of total 26291 rows for WEB reply with next pageToken 20000
fetched 10000 starting at 20000 out of total 274617 rows for WEB article with next pageToken 30000
fetched 6291 starting at 20000 out of total 26291 rows for WEB reply
no more rows to fetch for WEB reply
fetched 10000 starting at 30000 out of total 274617 rows for WEB article with next pageToken 40000
no more rows to fetch for WEB reply
fetched 10000 starting at 40000 out of total 274617 rows for WEB article with next pageToken 50000
no more rows to fetch for WEB reply
fetched 10000 starting at 50000 out of total 274626 rows for WEB article with next pageToken 60000
no more rows to fetch for WEB reply
fetched 10000 starting at 60000 out of total 274626 rows for WEB article with next pageToken 70000
no more rows to fetch for WEB reply
fetched 10000 starting at 70000 out of total 274626 rows for WEB article with next pageToken 80000
no more rows to fetch for WEB reply
fetched 10000 starting at 80000 out of total 274626 rows for WEB article with next pageToken 90000
no more rows to fetch for WEB reply
fetched 10000 starting at 90000 out of total 274626 rows for WEB article with next pageToken 100000
no more rows to fetch for WEB reply
fetched 10000 starting at 100000 out of total 274626 rows for WEB article with next pageToken 110000
no more rows to fetch for WEB reply
fetched 10000 starting at 110000 out of total 274626 rows for WEB article with next pageToken 120000
no more rows to fetch for WEB reply
fetched 10000 starting at 120000 out of total 274626 rows for WEB article with next pageToken 130000
no more rows to fetch for WEB reply
fetched 10000 starting at 130000 out of total 274626 rows for WEB article with next pageToken 140000
no more rows to fetch for WEB reply
fetched 10000 starting at 140000 out of total 274626 rows for WEB article with next pageToken 150000
no more rows to fetch for WEB reply
fetched 10000 starting at 150000 out of total 274625 rows for WEB article with next pageToken 160000
no more rows to fetch for WEB reply
fetched 10000 starting at 160000 out of total 274625 rows for WEB article with next pageToken 170000
no more rows to fetch for WEB reply
fetched 10000 starting at 170000 out of total 274625 rows for WEB article with next pageToken 180000
no more rows to fetch for WEB reply
fetched 10000 starting at 180000 out of total 274625 rows for WEB article with next pageToken 190000
no more rows to fetch for WEB reply
fetched 10000 starting at 190000 out of total 274625 rows for WEB article with next pageToken 200000
no more rows to fetch for WEB reply
fetched 10000 starting at 200000 out of total 274625 rows for WEB article with next pageToken 210000
no more rows to fetch for WEB reply
fetched 10000 starting at 210000 out of total 274625 rows for WEB article with next pageToken 220000
no more rows to fetch for WEB reply
fetched 10000 starting at 220000 out of total 274625 rows for WEB article with next pageToken 230000
no more rows to fetch for WEB reply
fetched 10000 starting at 230000 out of total 274625 rows for WEB article with next pageToken 240000
no more rows to fetch for WEB reply
fetched 10000 starting at 240000 out of total 274625 rows for WEB article with next pageToken 250000
no more rows to fetch for WEB reply
fetched 10000 starting at 250000 out of total 274625 rows for WEB article with next pageToken 260000
no more rows to fetch for WEB reply
fetched 10000 starting at 260000 out of total 274625 rows for WEB article with next pageToken 270000
no more rows to fetch for WEB reply
fetched 4625 starting at 270000 out of total 274625 rows for WEB article
fetched 10000 starting at 0 out of total 43122 rows for LINE article with next pageToken 10000
fetched 10000 starting at 0 out of total 30912 rows for LINE reply with next pageToken 10000
fetched 10000 starting at 10000 out of total 43122 rows for LINE article with next pageToken 20000
fetched 10000 starting at 10000 out of total 30912 rows for LINE reply with next pageToken 20000
fetched 10000 starting at 20000 out of total 43122 rows for LINE article with next pageToken 30000
fetched 10000 starting at 20000 out of total 30912 rows for LINE reply with next pageToken 30000
fetched 10000 starting at 30000 out of total 43122 rows for LINE article with next pageToken 40000
fetched 912 starting at 30000 out of total 30912 rows for LINE reply
no more rows to fetch for LINE reply
fetched 3122 starting at 40000 out of total 43122 rows for LINE article
fetched 10000 starting at 0 out of total 58616 rows for LIFF article with next pageToken 10000
fetched 10000 starting at 10000 out of total 58616 rows for LIFF article with next pageToken 20000
fetched 10000 starting at 20000 out of total 58616 rows for LIFF article with next pageToken 30000
fetched 10000 starting at 30000 out of total 58616 rows for LIFF article with next pageToken 40000
fetched 10000 starting at 40000 out of total 58616 rows for LIFF article with next pageToken 50000
fetched 8616 starting at 50000 out of total 58616 rows for LIFF article
fetched 10000 starting at 0 out of total 63351 rows for LIFF reply with next pageToken 10000
fetched 10000 starting at 10000 out of total 63351 rows for LIFF reply with next pageToken 20000
fetched 10000 starting at 20000 out of total 63351 rows for LIFF reply with next pageToken 30000
fetched 10000 starting at 30000 out of total 63351 rows for LIFF reply with next pageToken 40000
fetched 10000 starting at 40000 out of total 63351 rows for LIFF reply with next pageToken 50000
fetched 10000 starting at 50000 out of total 63351 rows for LIFF reply with next pageToken 60000
fetched 3351 starting at 60000 out of total 63351 rows for LIFF reply