search

coralproject / pillar

Deprecated: Service layer for the Coral ecosystem
Other
4 stars 1 forks source link

Integrate Pillar with Atoll #91

Closed impronunciable closed 8 years ago

impronunciable commented 8 years ago

To provide meaningful NLP stats for our users.

alexbyrnes commented 8 years ago

Readability scores are being generated with this: https://github.com/coralproject/atoll/pull/56

Reasons I like this integration better than doing it from the Pillar side:

1) reduces the number of http servers 2) the publisher can be something other than pillar. 3) pillar/atoll can work at different speeds and rabbitmq helps organize the data streams. 4) Atoll works as a library so we might as well take advantage.

I'd like to hear people's thoughts though since integration is a big deal for us.

gabelula commented 8 years ago

Yes. I think it makes sense to start integrating Atoll and do all the aggregation stuff that we are doing in Stats in Pillar with Atoll.

alexbyrnes commented 8 years ago

This example might be a good place to start. Gets average word counts a la Pillar.

Scaling this will depend on a variety of factors that I think we should sort out as soon as we can. Really (very) roughly speaking if a site gets 10k comments a day I think this will be fine.

alexbyrnes commented 8 years ago

42 aggregations that are currently in pillar.

alexbyrnes commented 8 years ago

These are in the coralproject/stats repo. Similar approach but keeping things that were meant to be done with the atoll library as a dependency out of atoll proper. The first stat that depends on the body of the comment is there now too. Feel free to give feedback/pr.