search

islamic-network / alquran.cloud

Web App that powers https://alquran.cloud
GNU General Public License v3.0
317 stars 44 forks source link

CDN sending null responses. #13

Closed elboboua closed 5 years ago

elboboua commented 5 years ago

Requests to the CDN to receive surah audio is not working with at least alafasy's audio files. Two days ago, it was working fine.

example request: https://cdn.alquran.cloud/media/audio/ayah/ar.alafasy/1 response: not found

meezaan commented 5 years ago

I am able to see it. Where are you based? Is the response still a 200?

elboboua commented 5 years ago

Salam, brother.

I am based in Lebanon.

It seems like the issue has been resolved and my chrome browser has cached the null responses from yesterday. Did you notice it being down yesterday or was I just experiencing issues on my end?

meezaan commented 5 years ago

Alaykum Salaam.

I did not see any alerts come through. Cloudflare did not report any outages on any of their PoPs either, so it is strange that your browser somehow cached a null response.

I will have a look in the logs to see if something happened and will let you know if I find anything.

elboboua commented 5 years ago

So it seems that there is still an ongoing issue in my region at the least. Only certain ayat are playing when I use them as a url in the browser, but none are triggering onload events because the responses are coming back null.

Screen Shot 2019-09-05 at 11 37 12 PM Screen Shot 2019-09-05 at 11 36 53 PM
meezaan commented 5 years ago

OK I've found the issue, I'm on it.

meezaan commented 5 years ago

So, the whole thing runs on a Docker Swarm between Digital Ocean and Linode with mounted volumes that are synchronised. On September 4, it appears that Digital Ocean restarted the VM (without telling us, they tend to do that - Linode is much better here) except this time they managed to bring it up with a volume with the files somehow being missing. That then synchronised with the Linode VM and deleted those files.

I've tried restoring from the backup at Digital Ocean spaces, but that times out to. So I'm now restoring from a backup from OneDrive. This will take time, so I appreciate your patience and thank you for reporting it.

I think it's also time to stop using Digital Ocean. We added them to the mix to diversify to the infrastructure, but they randomly restart machines and this has somehow been a result of that.

I'll keep you posted when the restore is done.

meezaan commented 5 years ago

The files have been restored now and I've flushed the cache at Cloudflare, so you should be able to get the files now after flushing your browser's cache.

Please let me know if you have any issues and thank you for bringing this to my attention.

elboboua commented 5 years ago

Much appreciated, Meezaan! Do you have a twitter I could follow to keep up with your work?

meezaan commented 5 years ago

Alhamdulillah. Sorry brother, I don't have a twitter account. Your thanks is enough.

You can join the Islamic Network Discord group - I do try to mention changes / updates there - https://discord.gg/FwUy69M.