search

openaustralia / oaf

OpenAustralia Foundation
https://github.com/openaustralia/oaf#readme
4 stars 3 forks source link

Disk IO is topping out (OAF-111) #110

Closed oajirabot closed 10 years ago

oajirabot commented 10 years ago

Over the last few days the disk IO has been occasionally maxing out which has caused intermittent outages of the sites.

At first I thought this was due to actions we were doing or cronjobs like OA's sitemap but we've started to see it spontaneously happen when nothing is apparently going on.

Issue details imported from Jira:

Detail Value
Assignee Unassigned
Reporter Henare Degan
Created 2012-11-07 14:27:11
Updated 2012-11-13 12:37:15

Imported Jira Comments

Henare Degan - 2012-11-07 14:32:50

I was seeing lots of Varnish in iotop so I decided to switch to memory storage from file storage as per best practices.

I made this change at 2012-11-07 14:04. So far New Relic tells me this has made performance worse. Disk IO utilisation and IO wait for the last 30 minutes are both noticeably up on the previous 30 minutes.

Henare Degan - 2012-11-07 14:44:52

Looking at utilisation over the last 6 hours I think it's fairer to say the change hasn't done anything good or bad.

It's only using 70M of memory so I guess that's the size of the cache at the moment (which is what I'd expect).

I'm still seeing the process doing lots of disk writes. What writes? Probably the log buffer /var/lib/varnish/kedumba/_.vsl, in fact this is probably what it was before anyway so my change won't have done much to impact disk IO (which is what the monitoring is telling us! Funny that).

Henare Degan - 2012-11-13 12:37:15

Our VPS provider advised this may have been due to another workload on the host our VPS resides on.