search

CVNRneuroimaging / infrastructure

Issue tracking, system documentation and configs for operations side of the neuroimaging core @ Atlanta VA CVNR / Emory University
3 stars 2 forks source link

Ahh! Rama down! #169

Closed simonero closed 8 years ago

simonero commented 8 years ago

@rrmm @kmcgregor123456

I was using rama and suddenly I can't get onto the server. Timeout issue on both X2GO and ssh. I can get on pano fine so it isnt my computer/connection. At Emory now, machine appears to be on but nothing on the screens (yes I clicked to make sure they are on).

I was running a few things at the time. Hoping I didn't kill it?!?! I'm here now so I can troubleshoot anything in front of the machine if necessary.

?!?!

rrmm commented 8 years ago

@rrmm @kmcgregor123456

I was using rama and suddenly I can't get onto the server. Timeout issue on both X2GO and ssh. I can get on pano fine so it isnt my computer/connection. At Emory now, machine appears to be on but nothing on the screens (yes I clicked to make sure they are on).

I was running a few things at the time. Hoping I didn't kill it?!?! I'm here now so I can troubleshoot anything in front of the machine if necessary.

if the keyboard doesn't respond (and for example, the capslock light doesn't toggle), i would power cycle it and see if it comes back up. (it can take some time).

rob

simonero commented 8 years ago

@rrmm

No caps light to test unfortunately....

That will kill all of the processing that I was doing, right? Is there anything I should try or wait on first to try to preserve that if possible?

rrmm commented 8 years ago

@rrmm

No caps light to test unfortunately....

That will kill all of the processing that I was doing, right? Is there anything I should try or wait on first to try to preserve that if possible?

numlock light?

is there disk activity? if it's just overloaded with a job you're running, then it may come back...eventually, but otherwise power cycling may be the only way to get it back which will kill all your processing.

rob

simonero commented 8 years ago

@rrmm

No numlock light to test. No light at all apparently, grrr.

How can I determine if there's disk activity?

It does make some noise as if it's doing something.

kmcgregor123456 commented 8 years ago

There is a HDD (hard disk drive) light that is on the front of the box. It flashes when the disk is taking I/O. If it is off or solid, then there's an issue.

Power cycling will kill your processes, but if they are running as zombies right now anyway... Best to Carl Poppa them...

Keith M. McGregor, PhD VA RR&D Atlanta CVNR Emory University, Department of Neurology 352.359.8084 http://www.varrd.emory.edu

From: simonero notifications@github.com Sent: Friday, April 22, 2016 4:14:13 PM To: CVNRneuroimaging/infrastructure Cc: Keith McGregor; Mention Subject: Re: [CVNRneuroimaging/infrastructure] Ahh! Rama down! (#169)

@rrmmhttps://github.com/rrmm

No numlock light to test. No light at all apparently, grrr.

How can I determine if there's disk activity?

It does make some noise as if it's doing something.

You are receiving this because you were mentioned. Reply to this email directly or view it on GitHubhttps://github.com/CVNRneuroimaging/infrastructure/issues/169#issuecomment-213574587

This e-mail message (including any attachments) is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this message (including any attachments) is strictly prohibited.

If you have received this message in error, please contact the sender by reply e-mail message and destroy all copies of the original message (including attachments).

simonero commented 8 years ago

Light was solid. Re-booting. Better now. How should I go about DXing what caused this? Could running too many processes cause this? Doesn't appear to be a memory shortage - I had some logs running ps and free every 30sec and according to the last entries I had 5G memory free, and if I go back a lil bit there's never less than a bit over 2G.

Also while we're on this.... you mentioned another drive I could mount for space. Sitting around here at the WMB? Where would I find this drive and is that still an option?