beta pages sometimes don't load -- or are excruciatingly slow to load when resuming session

[eplebel]

i've noticed this before for a while now, but the problem doesn't seem to happen consistently and so i haven't been able to figure out a set of conditions for this to be reproducible. however, here's a screenshot of various error messages that might help troubleshoot the situation:

Actually, the critical error is likely the 503 error message, because once things start working again (by just waiting a long time -- or logging out and logging back in, though even this latter strategy doesn't always seem to work), the other two error messages are still being displayed even though the page loads fine (though I guess these two other error messages should also be looked at eventually.)

This might be related to #94 (HTTP keep-alive doesn't seem to be working

[alexkyllo]

I noticed this too, and Heroku's docs explain that it's a timeout error, but I don't know the root cause yet. I suspect that this query is just taking too long at times: https://github.com/ScienceCommons/api/blob/master/app/controllers/articles_controller.rb#L41 I restarted the app through Heroku and that seemed to help, though.

[eplebel]

UPDATE: I think I've now figured out how to make this problem reproducible. If you browse away from the landing page for more than 30 minutes and then go back to it, you get the problem basically every time (or if you're idle on an article page for more than 30 minutes and then try to browse to landing page, it just hangs forever).

Once you're in this state, there seems to be only 2 ways to get around the problem:

1. Clear your browser's cache, close the page, and then browse back to landing page (INCONSISTENTLY gets around problem)
2. Initiate a search query, browse to an article page, and then go back to landing page (INCONSISTENTLY gets around problem)

UPDATE number 2: both of those actions sometimes does not resolve the problem (doh!)

[alexkyllo]

This change https://github.com/ScienceCommons/api/pull/230 seems to have helped significantly, the API calls to https://www.curatescience.org/articles/recently_added?limit=10 and https://www.curatescience.org/articles/recent?limit=10 from the landing page are taking 2-3 seconds in production for me now, whereas before they were taking 20-30+ and timing out. @eplebel try it out and let me know how it's performing for you.

At this point the query itself is pretty efficient--I may be able to make it a little more efficient by adding a sorted index so I'll try that next. But to improve performance much further beyond that, I think you'll need to upgrade the database plan on Heroku. The Hobby Basic postgres tier doesn't have an in-memory cache, so it's hitting the hard disk every time you refresh. If you upgrade to the Standard tier, postgres will be able to cache query results in RAM and serve them an order of magnitude faster. Here's the info on the plans: https://devcenter.heroku.com/articles/heroku-postgres-plans

[eplebel]

It's definitely improved to 2-3 seconds, and at least without timing out (though i still want to test a bit more before closing the issue). How much would it speed things up if we only had 5 instead of 10 articles loading?

thanks for the additional info regarding the caching option of the Standard tier; i'll keep that in mind.

[eplebel]

The page still seems to take 10+ seconds to load after an extended time of inactivity (i left my laptop on all night and in the morning observed this). However, I will confirm that this is reproducible tomorrow.

[alexkyllo]

Minifying main.js cut about 1.3 seconds off the loading time (for me). I just e-mailed you a little more information on that.

[alexkyllo]

I haven't seen the 503 gateway timeouts since I optimized the query for the landing page. That also speeded things up quite a bit, and minifying the JS helped too. But I don't think there's much else I can do in the code itself to speed the page load up further, so I think we can consider this issue closed for now.