search

ReactionMechanismGenerator / RMG-website

A Django-powered website for Reaction Mechanism Generator (RMG)
Other
21 stars 29 forks source link

Browsing database painfully slow on first load #185

Open rwest opened 4 years ago

rwest commented 4 years ago

This may be a consequence of some well-intentioned "lazy" loading (I remember programming something along this lines ages ago, to greatly accelerate booting the web server, which seemed a priority at the time of frequent development and debugging) but now often when I go to browse parts of the database, eg. https://rmg.mit.edu/database/kinetics/libraries/ it takes AAAAAGES to load the page (as it, presumably, loads the database into memory for the first time). With multi-threaded web server, each subprocess has to do this the first time it's called. It can be super frustrating when trying to use the website.

Writing this, I have deja vu memories. Perhaps it's largely done, and we just need to fetch https://rmg.mit.edu/database/load a few times (enough to hit each thread) whenever the server / service is restarted, or the database updated. (Or maybe fix that method so it actually does what it's supposed to do, because I'm not sure that it does right now)

Or on a cron job?

Anyway, any efforts to improve response times as a user would be appreciated!

mliu49 commented 4 years ago

Yeah, this is a big issue. In theory, we do fetch /database/load when we start the server for every single process, but it seems like it might not be working properly. There is also code which will reload the database if it detects that files have changed, which could also be malfunctioning and doing unnecessary reloading.

rwest commented 4 years ago

Yes, the unnecessary reloading crossed my mind too. I think it checks the timestamps of files?. I recall once having a setup where the networked file server's clock had drifted and was out of sync with the computer checking, so time stamps were always wrong; make would always rebuild everything.

xiaoruiDong commented 4 years ago

I am sorry for the inconvenience. As mentioned by @mliu, currently, there is a mechanism to check if a file is modified and reload the database, but it does not work properly. Even if no modifications has been made, it is still doing unnecessary reloadings. And thanks @rwest for the advice, timestamp check can be potentially problematic. I will do some diagnosis on that, and let you know if I find something.