Implement sorting of posts from multiple sources

[seanmturley]

Description

As of the completion of #159, the homepage is pulling 25 posts from 3 subreddits, and presenting the cards without any additional sorting i.e. each subreddit's posts are grouped together.

Subreddit candidates

• r/LandscapePhotography
• r/beautifultrees
• r/natureporn

See also: r/sfwpornnetwork, which includes a directory of nature related subreddits.

Approach

This may take a little experimentation to get an ideal mix and arrangement of posts from different subreddits. The following should be taken into consideration and executed as appropriate:

• [x] Selecting less posts from the lower quality subreddits. This is currently unspecified, so defaults to 25 posts. This should be able to be limited in a similar way to the getSubreddits query i.e. by appending "&limit=5" to the query string.
• [x] Applying an appropriate type filter to the requests (e.g. "top"). This is currently set to "hot" for all subreddits.
• [x] Sorting the posts to prioritize the "best" ones. This might require pulling more posts from each subreddit than required, then filtering e.g. the "top" posts. It's unclear to what extent this is possible when comparing between different subreddits. Edit: The best way to do this may be to calculate the "hotness" of each post, and order by that - this should also prevent "stale" posts from less active subreddits from sticking around too much. According to these two posts, the formula for "hotness" is: log(abs(score))+(created/45000). A comment in the first post suggests the + changes to a - if the score is negative.
• [x] The total number of posts can be flexible. In general, the site defaults to displaying 25 results for subreddits or searches, but it may make sense to expand this for the homepage, since it's the focus of the whole project.

Edit: The above features are all enabled, though further modifications to the exact setup may need to be made to tweak which posts are shown and in what order.

Issues

• [x] Pinned posts are included in addition to the specified limit (e.g. this post). These should probably be excluded completely, as they're normally not photo posts. Note this is controlled by a boolean in the stickied property. Edit: Selecting only the "hottest" posts solved this problem.
• [x] #162
• [ ] #163

[seanmturley]

I've expanded each query for the homepage to an object accepting the subreddit and a new limit option, allowing the number of posts pulled from each subreddit to be set independently.

In doing this, I noticed a few issues:

• Pinned posts are included in addition to the specified limit (e.g. this post). These should probably be excluded completely, as they're normally not photo posts.
• Posts with "slides" don't display the image (e.g. this post). Media handling should either be improved to display the image(s) in some way, or these should be excluded from the results.

Both of the above will be added to the main tasks as issue to fix.

[seanmturley]

Did some work on media (or lack thereof), and on using "hotness" as a way to filter and sort results. This has been incorporated into the task above.

[seanmturley]

Added a utility function to calculate a post's "hotness". This will be used for sorting results.

[seanmturley]

Implemented a new utility function to sort posts by "hotness", then select the "hottest" n posts, with n being specified in the query.

This appears to be working well, but the actual number of posts and individual subreddit limits may need tweaking.

It's worth noting that:

• Some "pinned" posts are still showing up, so they'll need to be removed another way
• Posts from smaller/less active subreddits seem to be less "hot", so there's not the level of mixing I was hoping for. This may be less of an issue once I add a few more subreddits into the mix. In any case, it probably is a decent reflection of the quality of the posts, it just doesn't showcase that there's any sorting going on, which is unfortunate. If really desired, I could attempt some form of scaling based on subreddit size, though this will need to be nonlinear, otherwise the biggest (i.e. r/EarthPorn) will be excessively demoted just due to its shear number of subscribers.

[seanmturley]

I've made a first pass at normalizing hotness based on subreddit size, and deemphasizing the importance of the recency of posting. This may need some tweaking, but I think it will at least allow some posts through from smaller subreddits.

[seanmturley]

Noting that I need to keep an eye on weather new posts from r/beautifultrees ever make it onto the homepage. If they don't, I may want to tweak the "hotness normalization" algorithm.

[seanmturley]

Following up from the previous comment - a post from r/beautifultrees has made it onto the homepage - it's 1 day old and has a score of 11. This took a while because that subreddit has quite low activity.

[seanmturley]

Noting again that occasional posts from r/beautifultrees are making it to the home page.

Overall, the filtering of posts is working pretty well. I think the quality of posts from different subreddits is a bit variable. I could consider increasing the number of posts pulled from e.g. r/EarthPorn (which are consistently good and all always make the home page), while decreasing the amount pulled from subreddits with lower quality (e.g. r/natureporn). This doesn't eliminate the issue, since hotness doesn't correspond directly with image quality, but would increase the prominence of better quality images on average.

[seanmturley]

I've added a few more subreddits from r/sfwpornnetwork to the homepage. Their quality is a bit variable, reinforcing the need for what I discussed earlier - increasing the posts pulled from subreddits with more consistent quality, and limiting the posts pulled from those subreddits where the quality is often poor. This will also have performance benefits, as the total number of posts requested from the API can be reduced - it's currently requesting 10 posts from 10 subreddits = 100 posts total, while only displaying the "hottest" 40.

[seanmturley]

I've adjusted the API call post limits for the homepage such that there are now three different tiers of subreddits:

• 20 posts: r/EarthPorn
• 10 posts: r/AutumnPorn, r/LandscapePhotography, r/natureporn, r/winterporn
• 5 posts: r/beautifultrees, r/desertporn, r/lakeporn, r/seaporn, r/waterporn

This provides an additional level of granularity for controlling the quality of posts that get through to the homepage.

The total number of posts requests is slightly reduced from the previous commit to 85, with the "hottest" 45 being displayed. The performance of this is not great - that's probably unavoidable to some extent due to making 10 separate API calls, but could perhaps be alleviated by:

• Reducing the number of posts requested and displayed
• Revisiting excess image quality (see #37)
• I have also created #168 to at least provide the user some visual feedback that the page is loading