kartoza / GeoHosting

https://kartoza.github.io/GeoHosting/
GNU Affero General Public License v3.0
0 stars 0 forks source link

GSH DR Strategy Drill #144

Open LokoMoloko98 opened 1 day ago

LokoMoloko98 commented 1 day ago

Simulate a Disaster Recovery execution on one of the GSH GeoNode instances by:

  1. Abruptly killing an instance that has a significant amount of Geospatial data that has been backed up via Velero
  2. Redeploy the same instance from the latest Velero back ups
  3. Have the GIS experts and DevOps monitor performance of the recovered GSH GeoNode instance
amyburness commented 21 hours ago

@Jeremy-Prior @seabilwe please check the current status. @jeremy @seabilwe we need to check that the restore was successful.

Jeremy-Prior commented 20 hours ago

@amyburness I'll check it out

Jeremy-Prior commented 19 hours ago

All resources back up on site

image

LokoMoloko98 commented 16 hours ago

The size2 GeoNode instance has undergone the DR drill and is back online. The former credentials are still in place. @seabilwe, @Jeremy-Prior, please check if all the data and configurations that was there before the DR are still in place.

Jeremy-Prior commented 2 hours ago

GeoNode is up and accounts work for signing in.

The resources are back (datasets, documentation, maps, geostories): image

Different permissions associated with the various resources were also restored: image

Raster data and documents appear to be restored and displaying correctly, however, vector data is not displaying when viewing maps or datasets. Example on dataset: image

Example on map (the map's annotations were restored correctly): image

@LokoMoloko98, to view the issue:

  1. Log in to the GeoNode
  2. Click view on a dataset. Dataset resources are symbolised by this image on the resource page
  3. Look at the visualisation and note only the backing map is displayed.
Jeremy-Prior commented 2 hours ago

Seems to be coming from the geoserver: image

Steps to get to error page:

  1. Log in to the GeoServer (can be accessed from the GeoNode using the Admin account)
  2. Click on layers on the lefthand side
  3. On Layers page, click on a layer title
  4. On the specific layer's page, click on the publishing tab
Jeremy-Prior commented 1 hour ago

Additionally, you can't log straight into the geoserver anymore, you need to go from the GeoNode to the GeoServer

geoserver-login

seabilwe commented 58 minutes ago

@LokoMoloko98 these are the recent full logs from geoserver geoserver.log

LokoMoloko98 commented 50 minutes ago

Thank's @seabilwe. The recovered version size2 is essentially a new deployment and as part of the deployment process credentials were created. From what @Jeremy-Prior and I picked up the creds that word are the:

The logs you posted show that the kartoza-geonode user is not being authenticated and it probably because its using the old creds