Very unstable loading

[lusiaold]

Hello, I'm using your API in my app, but it's very unstable. Sometimes JSON files are loading, sometimes not, sometimes only a few of all the JSON files are loading. Is there any way to fix this? Thanks

[Grayda]

Hi Pawel, I can take a look and work out what is going on. I have a possible idea of what could be going wrong, but I'll check. A few things:

1. What part(s) of the API are you using?

2. How often does it fail? I retrieve data every 2 minutes for push notifications and it's not often that I see an error

3. If possible, are you able to log the response when the API stops working? (e.g. just write what was received to a log file just for debugging purposes). If you run into a server error (e.g. too many requests per second or invalid coordinates sent for weather), it'll give you a JSON error back, and a 4xx or 5xx HTTP error code (e.g. 401, 404, 500 etc.). I'm keen to see what is being returned when it fails.

4. I think the issue might be an upstream issue with the Space Weather Prediction Centre, because I often get "cron failed" emails where the connection to SWPC has timed out or was closed by the remote side. I believe I've got code to detect that (and returned the cached copy of the data I process), but there might be a bug with that, so I'll need to check it.

So the tl;dr version is: I'm retrieving data from the SWPC but that fails sometimes. The API should serve you the last bit of cached information it has (which is no older than 2 minutes), but there might be a bug with that.

Thanks for reporting the issue, and I'm glad to see someone is using the API :)

[lusiaold]

Thanks for response! I'm using separately data for Kp level, Bz level, solar wind speed, and solar wind particles density. I can't tell you how often it fails, it's just irregular. What's strange, some JSONs are loading faster than other JSONs, one file can load immediately, for other one I must wait for even a minute. This bug is not only present in my app but also when I'm loading a JSON file in the browser - sometimes I must wait for a minute for a file to load :/

[Grayda]

It might be easier / better to combine the Kp, Bz, speed and density in one call, if you're not doing so already:

https://api.auroras.live/v1/?type=ace&data=all

Though you probably have a good reason for getting all the data separately.

JSON speeds can vary, because all the data is cached. The API looks at the Expires Header and caches the data until that time. So if you've just visited the site after someone else, it's going to load quicker because the info is already cached.

But with that said, the JSON shouldn't take minutes to load. I suspect this is part of point 4 I mentioned above, where the connection to the SWPC times out. I'll look and see if there's a way to make it time out MUCH sooner, and just return the cached data.

More info on caching can be found here (in the "Caching" section). I added it about 2-3 weeks ago

[lusiaold]

Ok, it seems that the lags are SWPC server-sided. I'll probably add an message in my app that loading the data sometimes may take some time. Anyways thanks for your help and have a nice day!

[Grayda]

No worries at all. I'll still add in the changes I mentioned so you get a more reliable service. Good luck with your app. I wouldn't mind seeing it when it's done :)

[lusiaold]

Thank you! If you want to see my app, then here's the link - https://play.google.com/store/apps/details?id=com.pawples.sdosun I'm planning to make cards better, with images. Unfortunately, I had to set minimum API level of my app to 21, which is android 5.0 lollipop, because the images just weren't loading on other devices. :/

[Grayda]

Looks really good so far!

I've just pushed a change to the API that adds a second level of fallback caching. It now works like this:

• If the item is cached, return it.
• If it's not cached (or has expired), re-fetch the item, cache it, store it in the database, and return it.
• If the item is not cached and attempting to re-fetch it times out after 10 seconds, return the last copy from the database
• If the item isn't cached, and re-fetching fails, and the item is not in the database, return an error

The timeout ensures issues at the SWPC don't hold everything up, and fetching it from the database ensures that if the SWPC does have issues, data is still returned.

Hopefully this provides a more reliable experience for you :)

[lusiaold]

Wow, I just tested activities with your API in my app! They are loading immediately now! Thanks very much for help! 😃

[Grayda]

No worries :)

The caching was already in place, and you probably loaded similar data to me while I was testing, so the load time would be almost instantaneous. As I mentioned, the cache is shared between all API users, so the more people that use it, the faster it'll be.

Good luck with your app!