Pipeline: Estimate maximum capacity for a build in travisci

[rivernews]

The build timeout is 50 minute.

Picking one build history as example:

• n=26 takes 1 min 43 sec
• n=129 takes 2 min 26 sec

Using the same proportion, 50 min, which is 3000 sec, can theoretically process 7186 reviews.

A quick estimation

• N=7000 is close to the upper bound
• N=6000 is safer, but still vulnerable to accidents.
• N=3000 is confident; if we can have some sort of chunk-processing mechanism then this is not a limit; otherwise we'll have to find other ways like cronjob on our own cloud.

[rivernews]

New travis build duration data point -

• Facebook, n=1510, duration: 23 min 35 sec .981. --> using a maximum of 50 minutes (which is 3000 seconds), n~=3000 reviews at max.
  • In the review parsing process, including switching to next page, the speed is about 1 sec per review.

We added additional logics, including javascript execution and slack logging, so that can slow things down.

[rivernews]

Updating data point

• At this point with the scraper version, we have 3203 reviews processed for 00:48:41.870.