kartoza / GeoHosting

https://kartoza.github.io/GeoHosting/
GNU Affero General Public License v3.0
0 stars 0 forks source link

GSH DR Strategy Drill #144

Closed LokoMoloko98 closed 2 months ago

LokoMoloko98 commented 2 months ago

Simulate a Disaster Recovery execution on one of the GSH GeoNode instances by:

  1. Abruptly killing an instance that has a significant amount of Geospatial data that has been backed up via Velero
  2. Redeploy the same instance from the latest Velero back ups
  3. Have the GIS experts and DevOps monitor performance of the recovered GSH GeoNode instance
amyburness commented 2 months ago

@Jeremy-Prior @seabilwe please check the current status. @jeremy @seabilwe we need to check that the restore was successful.

Jeremy-Prior commented 2 months ago

@amyburness I'll check it out

Jeremy-Prior commented 2 months ago

All resources back up on site

image

LokoMoloko98 commented 2 months ago

The size2 GeoNode instance has undergone the DR drill and is back online. The former credentials are still in place. @seabilwe, @Jeremy-Prior, please check if all the data and configurations that was there before the DR are still in place.

Jeremy-Prior commented 2 months ago

GeoNode is up and accounts work for signing in.

The resources are back (datasets, documentation, maps, geostories): image

Different permissions associated with the various resources were also restored: image

Raster data and documents appear to be restored and displaying correctly, however, vector data is not displaying when viewing maps or datasets. Example on dataset: image

Example on map (the map's annotations were restored correctly): image

@LokoMoloko98, to view the issue:

  1. Log in to the GeoNode
  2. Click view on a dataset. Dataset resources are symbolised by this image on the resource page
  3. Look at the visualisation and note only the backing map is displayed.
Jeremy-Prior commented 2 months ago

Seems to be coming from the geoserver: image

Steps to get to error page:

  1. Log in to the GeoServer (can be accessed from the GeoNode using the Admin account)
  2. Click on layers on the lefthand side
  3. On Layers page, click on a layer title
  4. On the specific layer's page, click on the publishing tab
Jeremy-Prior commented 2 months ago

Additionally, you can't log straight into the geoserver anymore, you need to go from the GeoNode to the GeoServer

geoserver-login

seabilwe commented 2 months ago

@LokoMoloko98 these are the recent full logs from geoserver geoserver.log

LokoMoloko98 commented 2 months ago

Thank's @seabilwe. The recovered version size2 is essentially a new deployment and as part of the deployment process credentials were created. From what @Jeremy-Prior and I picked up the creds that word are the:

The logs you posted show that the kartoza-geonode user is not being authenticated and it probably because its using the old creds

LokoMoloko98 commented 2 months ago

@Jeremy-Prior, @seabilwe all good for closing?

cc: @lgkgh, @amyburness

Jeremy-Prior commented 2 months ago

@LokoMoloko98, yes. Everything is back online and th layers are appearing. Additionally signing in and out of geonode and the geoserver works correctly.