The way we store reviews - as separate, fragment individual file in S3 is just optimized for scraper.
However, in order to do data-dense analytics, data structure needs to come in chunk in order to scale. This duality can be complete by adding a cronjob which goes across all orgs and aggregate their review objects into a single tsv file, which is optimized for data-dense operation.
This is likely to get support from slack middleware service.
The way we store reviews - as separate, fragment individual file in S3 is just optimized for scraper.
However, in order to do data-dense analytics, data structure needs to come in chunk in order to scale. This duality can be complete by adding a cronjob which goes across all orgs and aggregate their review objects into a single
tsvfile, which is optimized for data-dense operation.This is likely to get support from slack middleware service.