search

frazer-lab / cluster

Repo for cluster issues.
1 stars 0 forks source link

SDSC Having Another Power Outage #256

Closed tatarsky closed 6 years ago

tatarsky commented 6 years ago

You are advised that SDSC has reported another power outage. Unclear of full impact but connectivity to the Cluster is down and we may be looking at a full drain/reboot again if its like the last one.

//

The SDSC Datacenters have experienced some level of power outage. Cause of this, as well as impacts, are currently being assessed and Facilities Management has been contacted. Updates will be provided at status.sdsc.edu as we have them.

Regards,

--Jeff

//

I will report any updates I get but you can also monitor:

http://status.externsdsc.org/

tatarsky commented 6 years ago

Its a full outage. When confirmed that its back on I will start the recovery process and I'm going to ask @kafrazer review the terms of the agreement with SDSC COLO. This is too many outages.

http://status.externsdsc.org/2018/10/05/sdsc-power-outage/

tatarsky commented 6 years ago

Power back.

//

Power has been restored to the building and all systems are back online, however systems backed by UPS-only service (no generator) are in bypass. A CPT technician is en route – ETA 0500 – to restore battery protection.

//

We are not on UPS protected racks. So the above doesn't make our situation less reliable to wait so I will start the Lustre recovery. Expect it to take awhile. I will update here when I am ready to re-open the systems.

tatarsky commented 6 years ago

I believe I have all items back running except Mysql which I usually leave for @hirokomatsui to restart but I can also get it.

Please report any items not restarted properly to this Git and I'll track them down.

tatarsky commented 6 years ago

We believe this incident is resolved. Please re-open if you find something still wrong.

kafrazer commented 6 years ago

Thank you.

From: Paul Tatarsky notifications@github.com Reply-To: frazer-lab/cluster reply@reply.github.com Date: Friday, October 5, 2018 at 11:59 AM To: frazer-lab/cluster cluster@noreply.github.com Cc: "Frazer, Kelly A." kafrazer@ucsd.edu, Mention mention@noreply.github.com Subject: Re: [frazer-lab/cluster] SDSC Having Another Power Outage (#256)

Closed #256https://github.com/frazer-lab/cluster/issues/256.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/frazer-lab/cluster/issues/256#event-1887888640, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AIRblvju2cOGWRfzX2wkb0kvvte5Zd71ks5uh6wtgaJpZM4XJ28F.