search

galaxycomputationalchemistry / usegalaxy-za

Landing page for info about Galaxy CompChem ZA
https://galaxycomputationalchemistry.github.io/usegalaxy-za/
0 stars 0 forks source link

ilifu Galaxy is down #1

Closed chrisbarnettster closed 3 years ago

chrisbarnettster commented 4 years ago

504 Gateway Time-out nginx/1.14.0 (Ubuntu)

Checked https://www.isitdownrightnow.com/galaxy-compchem.ilifu.ac.za.html and https://downforeveryoneorjustme.com/galaxy-compchem.ilifu.ac.za

Access via SSH/ Openstack is down.

chrisbarnettster commented 4 years ago

soft rebooting instance via Openstack

chrisbarnettster commented 4 years ago

soft reboot did not have desired effect. Choosing to shut off instance. System message is "Unable to shutoff instance"

Contact ilifu support. Issues with Ceph.

chrisbarnettster commented 4 years ago

Yesterday's email - The ilifu research cloud services are currently unavailable. The technical team is investigating the issue and is working to bring the services back up.

Today's email - The ilifu research cloud services are online and active.

galaxy-compchem is still down so investigating now

chrisbarnettster commented 4 years ago

image

chrisbarnettster commented 4 years ago

soft rebooting

chrisbarnettster commented 4 years ago

Please confirm your selection. Restarted instances will lose any data not saved in persistent storage. This is probably why it is not coming up. WIll attempt a hard stop and start, else I will revert to a save snapshot (Aug 2019) or rebuild.

chrisbarnettster commented 4 years ago

Tainted the galaxy instance and forced a terraform rebuild. Still have to rebuild the Galaxy and tools with ansible.

chrisbarnettster commented 4 years ago

Made some progress. Checked data all good so far and started a quick archive.

Base OS drive stuck and no access. Looks like the cloud provider CEPHFS is stalling.

chrisbarnettster commented 4 years ago

running playbook for reinstall.