search

AtlasOfLivingAustralia / spatial-hub

New spatial portal front end using AngularJS
https://spatial.ala.org.au/
7 stars 16 forks source link

Setup replacement for ala-maps/ala-maps-db #54

Closed adam-collins closed 6 years ago

adam-collins commented 6 years ago

This will be the new production server. There is a requirement to include the existing database and all analysis output.

ansell commented 6 years ago

Will the replacement be one or two servers? Ie, postgres and tomcat on the same or different servers.

ansell commented 6 years ago

Planning notes for the migration are being kept at:

https://wiki.ala.org.au/wiki/Spatial-systems

ansell commented 6 years ago

How does http://ala-dylan.it.csiro.au/sampling-service fit into this task?

ansell commented 6 years ago

The disk usage on the current servers (to commission replacements) is:

ans025@ala-maps:~> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3             266G  124G  129G  49% /
udev                   63G  164K   63G   1% /dev
/dev/sda1             479M   34M  421M   8% /boot
/dev/sdb1             985G  651G  284G  70% /data
/dev/sdc1             2.0T  1.2T  764G  61% /data2
/dev/sdd1             2.0T  1.6T  356G  82% /data3
ans025@ala-maps-db:~> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3             132G   60G   66G  48% /
udev                   32G  120K   32G   1% /dev
/dev/sda1             479M   34M  421M   8% /boot
/dev/sdb1             985G  245G  690G  27% /data

The memory usage on the current servers (at this point in time, noting that swap is being actively used on ala-maps) is:

ans025@ala-maps:~> free -m
             total       used       free     shared    buffers     cached
Mem:        128805     122741       6064          0       6053      25360
-/+ buffers/cache:      91327      37478
Swap:         2047       2029         17
ans025@ala-maps-db:~> free -m
             total       used       free     shared    buffers     cached
Mem:         64294      29527      34766          0        619      27600
-/+ buffers/cache:       1307      62987
Swap:         2047          0       2047

The cpu specs are:

ans025@ala-maps:~> lscpu
Architecture:          x86_64
CPU(s):                32
Thread(s) per core:    2
Core(s) per socket:    4
CPU socket(s):         4
NUMA node(s):          4
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 46
Stepping:              6
CPU MHz:               1861.996
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              18432K
ans025@ala-maps-db:~> lscpu
Architecture:          x86_64
CPU(s):                16
Thread(s) per core:    2
Core(s) per socket:    4
CPU socket(s):         2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 46
Stepping:              6
CPU MHz:               1861.994
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              18432K
matthewandrews commented 6 years ago

Thanks. What I had sketched in for spatial prod at AWS was an r4.2xlarge instance (8 cpu, 60g memory) with 2T storage.

ansell commented 6 years ago

That would be a drastic reduction in all of the servers resources. Has there been a performance analysis that recommended such drastic reductions or is it purely for cost savings? Only having 2T of storage may not work for what Adam and I have planned so far. 60G of memory is also a drastic reduction compared to the current setup, but there are a different set of web applications that will be deployed so it may or may not work. The current server is using most of its 128G and all of its 2GB swap to the extent that I cannot run du or grep without having it crash one or another of the tomcat instances due to OutOfMemoryException. Going from 32 cores down to 8 may have an effect on what we can do with the new server, but I am not familiar with why the current server has 32 or how often that becomes a bottleneck in our systems.

What is planned so far for the data migration (documented at the wiki https://wiki.ala.org.au/wiki/Spatial-systems ) is basically to copy everything from /data/ala (which includes symlinks to /data3), and /data2/ala, across to the replacement system and have it automatically transformed by spatial-hub and geoserver. That copy operation alone will not complete with only 2T of storage unless there are some drastic deletions or large omissions that can be done. Even after the copy completes, there may be temporary files created by the transformation process that can be much larger than the existing files or the final transformed files, based on working with GDAL and spatial layers in the past.

I have a large number of spatial layers on my loading list, including the 2016 Census data and a large number of raster layers, which have been blocked by the solr cloud and cassandra infrastructure work for 12+ months, but if storage isn't available on the new spatial servers, they will be blocked into the future by this as well.

Also, was the plan to have the postgres instance also hosted on the same machine or locate it on a dedicated machine as it is currently?

matthewandrews commented 6 years ago

That was just a rough guess based on Adam saying we need something similar to spatial-test but with more memory and CPU ... and assuming that postgres was on the same instance.

My impression was that the new version of spatial would be significantly more efficient in resource use...?

What is a more suitable set of specs?

Re db, another option, if we are using Amazon for production, is to use their RDS Postgres. Or even the AWS Aurora postgres-equivalent db, which I haven't used:

Amazon Aurora PostgreSQL-compatible edition Enterprise-class database, starting at <$1/day. Up to 3 times the throughput of PostgreSQL. Up to 64TB of auto-scaling SSD storage. 6-way replication across three Availability Zones. Up to 15 Read Replicas with sub-10ms replica lag. Automatic monitoring and failover in less than 30 seconds.

adam-collins commented 6 years ago

The sampling-service is separate. There are no plans to update it at this time.

Tasilee commented 6 years ago

The number of 'environmental' layers must grow (if we want to remain relevant and leading) so we need a strategy to ensure efficient ingestion/processing, analytical support (layers are more for analysis than viewing), and metadata exposure. Longer term strategy is for a more suitable agency to host/manage. Intermediate strategy is a standard point-intersect web service (single and bulk) for some agency to coordinate.

ansell commented 6 years ago

What I have to work on for a recommendation so far is that the current memory (128GB) is too small on ala-maps. However, I am not sure what memory requirements the future production load with the different new web applications will require so that one is an open question still. If it is really more than twice as efficient that would be great but unless anyone has replayed the queries that are currently being pushed through production, twice as memory efficient as an assumption seems very risky.

The disk space in total currently being used is around 4TB out of the 5TB currently available on ala-maps. However, because of the memory limitations, I have not been able to run du successfully to break that down further without having one of the tomcat instances crash, so I can't be more specific than the general df statistics above. I haven't looked into the data transformations that are going to be run to transform the data, so I am not sure what the additional temporary requirements will be.

One of the steps is to let the new version of geoserver transform its data directory itself, which I have no experience with, and don't want to be repeating multiple times if at all possible.

Adam has given me a compression command using gdal for some of the current layers before copying them off ala-maps, but I haven't looked into scripting that yet or what the gains will be in terms of memory. Given I am intentionally not running du on the current server since it is in production, I will need to figure out another way using ls or something like that to see if I can gauge the disk size improvements before and after running that command.

In terms of new layers that we have to get back to loading once the infrastructure changes are complete, there are both raster (large disk space) and vector (small disk space) layers in the planning.

Based on the irregular performance that RDS MySQL had for the specieslists application, I don't think there is enough time to debug new failures relating to the use of RDS Postgres in the current spatial sprint. I have also spent a large amount of time this year unsuccessfully working through AWS Aurora query performance/failure issues with the AEKOS data team, so I have a bad impression of it currently.

I haven't had experience with Postgres+Postgis on ala-maps-db, but running that combination locally it generally seems to be CPU-bound, rather than IO-bound. The issue of possibly not being able to manage our Postgres+Postgis versions (and having to retest the applications with whatever is currently supported there) if we move to either RDS or Aurora are also questions that would require more time to investigate and debug than we have in the current sprint. One issue with guessing the performance of Postgres+Postgis is that we have been holding off on loading any new spatial layers for over a year except where absolutely necessary, so it is difficult to know whether the 16 cores on ala-maps-db are sufficient for what we need.

My experience with Postgres/Postgis performance has mostly been with spatial-test (soon to be reborn as nectar-spatial-staging), which was both heavy on CPU and IO during new layer loads, to the point that some of the more complex vector layer joins took so long that we had to either wait days/weeks for completion or cancel them. However, spatial-test is substantially less resourced compared to ala-maps-db, so it is possible the useful range for loading new layers is somewhere between the 4 core/30 GB on spatial-test and the 16 core/64 GB on ala-maps-db.

adam-collins commented 6 years ago

Storage requirements for the production replacement, SSD is not required.

An initial ala-maps replacement with 128G memory and 16x cpu is acceptable. Once stable this can be replaced with smaller master and slave instances.

ala-maps-db can be absorbed by ala-maps. Postgres storage requirements are much smaller. Usage of an external solution is worth exploring.

matthewandrews commented 6 years ago

possibly not being able to manage our Postgres+Postgis versions (and having to retest the applications with whatever is currently supported there)

For info, this is the range of Postgres versions currently available on RDS: screen shot 2017-11-23 at 12 06 45 pm

matthewandrews commented 6 years ago

OK, I've put in the server inventory (AWS plan) that we will have an r4.4xlarge (16 cpu, 122g memory) instance with 4T of HDD storage (later reduced to 2T).

ansell commented 6 years ago

I think we are at the stage where we need to have the new server created to start configuring it and copying data across to test out whether I have identified everything that needs to be transferred.

@matthewandrews can you create the server and I will start installing software on it.

Not sure what ansible script will be the most ideal starting point as both of the two candidates (layers-service and spatial-portal) are broken right now for different reasons but I will work on those locally today while the server is requisitioned.

ansell commented 6 years ago

WIP pull requests for ansible at:

https://github.com/AtlasOfLivingAustralia/ala-install/pull/148

https://github.com/AtlasOfLivingAustralia/ansible-inventories/pull/14

Have to add a new role for geonetwork.

Also found a bug with multiple webapps and nginx only picked up one of them. Pretty sure I have had the same issue with apache before, just required copy-paste in the nginx sites-enabled/spatial.ala.org.au file to add the other servlets

ansell commented 6 years ago

Just for future reference, the specs on the current candidate for replacing ala-maps/ala-maps-db is:

ans025@aws-spatial-prod:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             60G     0   60G   0% /dev
tmpfs            12G  8.7M   12G   1% /run
/dev/xvda1      194G  132G   63G  68% /
tmpfs            60G  4.0K   60G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            60G     0   60G   0% /sys/fs/cgroup
/dev/xvdg       2.0T   71M  1.9T   1% /data-temp
/dev/xvdf       2.0T   78M  1.9T   1% /data
tmpfs            12G     0   12G   0% /run/user/1009
ans025@aws-spatial-prod:~$ free -m
              total        used        free      shared  buff/cache   available
Mem:         122878        4073      117848          20         957      118012
Swap:        131071           0      131071
ans025@aws-spatial-prod:~$ lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                16
On-line CPU(s) list:   0-15
Thread(s) per core:    2
Core(s) per socket:    8
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 79
Model name:            Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz
Stepping:              1
CPU MHz:               1199.234
CPU max MHz:           3000.0000
CPU min MHz:           1200.0000
BogoMIPS:              4600.14
Hypervisor vendor:     Xen
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              46080K
NUMA node0 CPU(s):     0-15
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx xsaveopt
ansell commented 6 years ago

Still to do:

Strictly ordered tasks during scheduled downtime

ansell commented 6 years ago

I was not aware until yesterday that in addition to the static layers and spatial portal user sessions, there are novel data points from biocollect being recorded into ala-maps-db and/or ala-maps and not stored anywhere else.

My current strategy was to open up a maintenance period where the rsync processes run and then there is a multi-day period where the session data would have been lost, on the basis that it isn't generally critical.

This is necessary because of the underprovisioning of storage on aws-spatial-prod, which requires directory reorganisation and the compression to both be successful to drop the storage from 5TB on ala-maps/ala-maps-db to less than 2TB on aws-spatial-prod. The current structure with 2x2TB storage attached does not allow enough free space for /data3, so I have not yet attempted to copy it across.

An alternative if it is possible to attach another 2TB volume to aws-spatial-prod could be to keep the current 3x2TB structure from ala-maps and leave the compression/deletion/reorganisation until after the migration is complete. Does the budget allow for another 2TB of storage to be attached to aws-spatial-prod to complete the migration earlier?

If that is possible, then the rsync processes can continue to run up to the date when switchover occurs, with only the time required to dump and restore the postgres database (~6-8 hours, but possibly longer due to the low AWS IOPS setup for aws-spatial-prod). This would ensure that there would be no lost database data as the scheduled downtime would ensure that operations that are using ala-maps/ala-maps-db as a read-write database would fail rather than succeed.

We are moving from Postgres-8.3 on ala-maps-db (or 8.4 if IM&T don't undo their automated SLES11 upgrade) to Postgres-9.6 on aws-spatial-prod, so it isn't possible to set up a master-slave environment to replicate the database automatically, hence why there still needs to be scheduled downtime for the switchover.

ansell commented 6 years ago

Scratch that, I am not even remotely confident that I understand where the data is located and what software stacks are interacting/writing-to/upgrading data structures to only do the postgres migration during the scheduled downtime. Once we have the processes working, we will need to wipe aws-spatial-prod clean and start the dumps and the rsync's again during the shutdown.

adam-collins commented 6 years ago

Restart setup of aws-spatial-prod without user layers or other analysis output.

These can be added later on if required.

ansell commented 6 years ago

What does that practically mean for the migration process? I am not familiar with the software or data structures enough to interpret the difference between user layers/analysis/other.

adam-collins commented 6 years ago

Closing because the server the old data on disk, if not yet available to spatial-service.