search

illinois-scicomp / machine-shop-maintenance

Scripts and Issues for the birds and the beers
10 stars 3 forks source link

[lager] Disk array is slow #52

Closed inducer closed 2 years ago

inducer commented 3 years ago

Something is wrong with the disk array. I don't know what. It's super slow, and has been ever since the maintenance on October 12. See also #51, #49 etc.

inducer commented 3 years ago 
/opt/MegaRAID/perccli# ./perccli /c1/e12 show phyerrorcounters 
------------------------------------------------            
EID PNO InDwdCnt RnDiErCnt LsDwSyCnt PRstPrbCnt             
------------------------------------------------                                                                        
 12   4        0         0         0          0                                                                            
(snip)
 12  34      236       233         2          0             
 12  35        7         7         1          0                                                                         
(snip)
------------------------------------------------   
 ./perccli /c1/e25 show phyerrorcounters
Controller = 1
Status = Success
Description = None

Detailed Status :
===============

------------------------------------------------
EID PNO InDwdCnt RnDiErCnt LsDwSyCnt PRstPrbCnt 
------------------------------------------------
 25   4        8         8        30          0 
 25   5       64       107        30          0 
 25   6      101       100        30          0 
 25   7       16        15        30          0 
(snip)
 25  27        0         0         0          0 
 25  28     2608      2504         1          0 
(snip)
------------------------------------------------

Maybe it's the enclosures, not the disks that are bad?

inducer commented 3 years ago

It seems to have recovered somewhat? I'm about to have a fairly stressful two weeks, so I'd be grateful if I could put off fixing it for a bit.

inducer commented 3 years ago

However:

One thing I have done is made a backup though So that might explain why it feels fast All the file metadata is probably still in the cache If I rebooted lager, it would probably revert to being slow

inducer commented 3 years ago

So, in summary, I'm not sure.

inducer commented 3 years ago

That seems to be something that self-resolves after a while, maybe after the caches warm up? Which? Not sure. Maybe the ones on the controller.

inducer commented 2 years ago

Not currently an issue, closing.