search

glamod / glamod-ingest

Database preparation and ingestion for GLAMOD
BSD 2-Clause "Simplified" License
2 stars 1 forks source link

Write a script to cluster (sort) all the partitions #58

Closed agstephens closed 3 years ago

agstephens commented 3 years ago

Can be a simple script to operate on a year range for a given domain and frequency.

agstephens commented 3 years ago

Script now lives here:

https://github.com/glamod/glamod-ingest/blob/master/scripts/manage/sort-db-tables.sh

Typical usage:

nohup ./scripts/manage/sort-db-tables.sh r2.0 land 0 > cluster.r2.0.land.0.txt &
nohup ./scripts/manage/sort-db-tables.sh r2.0 land 2 > cluster.r2.0.land.2.txt &
nohup ./scripts/manage/sort-db-tables.sh r2.0 land 3 > cluster.r2.0.land.3.txt &
nohup ./scripts/manage/sort-db-tables.sh r2.0 marine 0 > cluster.r2.0.marine.0.txt &

Check latest progress with:

$ for i in  cluster.r2.0.* ; do echo "----:  $i" ; tail -4 $i ; done
----:  cluster.r2.0.land.0.txt
CLUSTER
ENDED: Thu 11 Mar 18:15:01 GMT 2021
[INFO] Clustering (sorting): lite_2_0.observations_1978_land_0
STARTED: Thu 11 Mar 18:15:02 GMT 2021
----:  cluster.r2.0.land.2.txt
[INFO] Clustering (sorting): lite_2_0.observations_2020_land_2
STARTED: Thu 11 Mar 17:06:24 GMT 2021
CLUSTER
ENDED: Thu 11 Mar 17:06:28 GMT 2021
----:  cluster.r2.0.land.3.txt
CLUSTER
ENDED: Thu 11 Mar 18:28:10 GMT 2021
[INFO] Clustering (sorting): lite_2_0.observations_1926_land_3
STARTED: Thu 11 Mar 18:28:11 GMT 2021
----:  cluster.r2.0.marine.0.txt
CLUSTER
ENDED: Thu 11 Mar 18:30:53 GMT 2021
[INFO] Clustering (sorting): lite_2_0.observations_1912_marine_0
STARTED: Thu 11 Mar 18:31:01 GMT 2021