Improving the speed of the site

[Grayda]

The current API / website is hosted on shared hosting in the US. Static sites are generally pretty snappy, but due to the highly dynamic content (that changes every minute), and issues outside of our control, the API (which powers all the Auroras.live apps) can be slower from time to time.

For reference, the API pulls in the following data. Not all of this data is grabbed, as the client can change what data is obtained, but this can potentially include

• Weather from http://met.no (current weather, and / or 9 day forecast). This is not cached yet.
• Three day Kp forecast, 27 day Kp forecast, Bz, speed and density from SWPC (these 5 data points are from 4 separate sources)
• "Nowcast" probability percentage from our own database (that is updated every 10 minutes)
• Information about our archives (start date, total size etc.)

As there are 7 different points to download from internal and external sources, process and serve, the average request time is about 5 seconds from the request coming in, to the data being served.

Currently we're researching a few solutions that can be put in place to speed up the process, including:

• Better caching. Some data is cached, and all database calls are cached. The cache doesn't last very long, as some data (e.g. Bz) is updated every minute
• Migrating to a new server. We're investigating Google App engine, but due to the sheer number of database calls made every 10 minutes (500,000 rows as we download, parse and import all Nowcast data), the cost would be very high
• Having two separate back-end sites: One for database calls (to keep cost down) and one for preparation and sending of the data

Note that these are merely ideas at this stage. This issue is open to maintain transparency to users and to foster suggestions for improving site speeds.

[Grayda]

To further add to this:

Caching has been re-done on the API. As the changelog explains, all external data is cached for as long as the remote server specifies (so if met.no sets an Expiry header for 10 minutes from now, the cached version is served until then, then is fetched again).

Database calls are generally cached for 60 seconds, but archived data never expires unless I manually force it to expire (which probably won't happen, as this data will likely never change).

And just to clarify, data is cached per SQL query or remote URL. So for example http://example.com?id=0 and http://example.com?id=1 are two separate lots of cached data, while SELECT * FROM sometable WHERE id = 0 and SELECT * FROM sometable WHERE id = 1 are also two separate lots of cached data.

[Grayda]

I'm closing this, as we're now caching most data, and v2 of the API will take this further by caching things such as geolocation API calls, weather calls and such.

In addition, some data that was already being cached is being cached for longer periods of time, plus v2 will include scripts that run in the background to periodically refresh data so that calls are a lot quicker.