Rearchitect Script for 14.4x Speedup in Reactions Scraping

[minimaxir]

Scraping reactions is relatively slow for large pages (15 minutes for CNN's FB page) and will get worse as time goes by.

For example, when scraping 100 Statuses:

Current Architecture

• Return Post Metadata for All 100 Posts [1 HTTP Request]
• For each post, Query All Reactions (6) [100 HTTP Requests]

The query occurs during processing of the post so no extra data manipulation is necessary.

Better Architecture

• Return Post Metadata for All 100 Posts [1 HTTP Request]
• For all queried posts, retrieve a single reaction count. Repeat for each reaction [6 HTTP Requests]

The Reaction output from each of the 6 vectors must be mapped to the corresponding post.

101/7 = 14.4x speedup in HTTP, which is the bottleneck.

The challenge is implementing the mapping in a way that is easy to read. Tracking progress with this issue.

[baditaflorin]

Can we help ?

[minimaxir]

Done in https://github.com/minimaxir/facebook-page-post-scraper/commit/ba26e7797f93a55e8d35f7b7408e1c69ccde16ab

Actual speedup is like maybe 4-5x. Have not done benchmarks.