Data pipeline: cronjob-kind-of mechanism to track each org over time

[rivernews]

We need some kind of cronjob service support, likely from slack middleware service, to keep track of org data for the long term, so that analytics work can do time series.

[rivernews]

Pubsub issue

We got a fundamental problem. Our scraper running in travis, cannot connect to redis. Because our redis cluster currently only allows local (from k8) connections, our laptop (port-forwarding) connection, if we're keep using travis, we cannot use redis w/o opening it up to a hostname, or we have to install doctl and kubectl in travis (or at least run tf-k8-toolkit docker image as a on-the-fly container command --> but do remember there might be some more latency).

[rivernews]

Update Progress

• [ ] Provide periodic report of progress to slack md svc. Use fixed time interval, instead of quadric/five-quantile reporting to avoid timeout on the slack md svc end.