search

praw-dev / prawtools

A collection of tools that interact with reddit's API providing moderator utilities, subreddit statistics, and keyword alerts.
BSD 2-Clause "Simplified" License
115 stars 25 forks source link

Reduce Memory Consumption #39

Closed bboe closed 7 years ago

bboe commented 8 years ago

Stats bot, which continuously runs subreddit_stats, was consuming over 4.6GB of memory. That's too much. Rather than saving all the submissions and comments for later aggregation, they should be processed and discarded immediately.

bboe commented 7 years ago

This doesn't appear to be as big of an issue, but it is still problematic. After running for a few days it's consuming over 350MB.

danthedaniel commented 7 years ago

Why not utilize sqlite? It would also allow for SQL to be written for the analytics.

bboe commented 7 years ago

I have no desire to persist the results for the purposes of this tool.

bboe commented 7 years ago

This appears to now be resolved:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                    
12599 ec2-user  20   0  244m  51m 5432 S  0.0  5.2  84:12.11 statsbot