Open agstephens opened 3 years ago
Try running it for a single year and batch:
$ head -2 /gws/nopw/j04/c3s311a_lot2/data/level2/land/r202005/batches/cdmlite_batch_rules.txt
path_prefix|batch_id|of_n_batches|batch_length
/gws/nopw/j04/c3s311a_lot2/data/level2/land/r202005/cdm_lite/daily/CDM_lite_SecondRelease_ACW00011*|daily-CDM_lite_SecondRelease_ACW00011|5567|2
So let's run with:
./scripts/land/restructure-land.py -r r2.0 -b daily-CDM_lite_SecondRelease_AF000040 -y 1975
This writes output such as:
[INFO] Loading file list...
[INFO] Reading: /gws/nopw/j04/c3s311a_lot2/data/level2/land/r202005/cdm_lite/daily/CDM_lite_SecondRelease_AF000040930.psv.gz to detect years.
[INFO] Working on 1975 for: daily-CDM_lite_SecondRelease_AF000040
[INFO] Reading input files: /gws/nopw/j04/c3s311a_lot2/data/level2/land/r202005/cdm_lite/daily/CDM_lite_SecondRelease_AF000040930.psv.gz , etc.
[INFO] Lengths of data frames before filtering years: [6283]
[INFO] Lengths of data frames AFTER filtering years: [241]
[INFO] Writing output file: /work/scratch-nopw/astephen/glamod/r2.0/cdmlite/prepare/land/3/1975/3-1975-daily-CDM_lite_SecondRelease_AF000040.psv
[INFO] Wrote: /work/scratch-nopw/astephen/glamod/r2.0/cdmlite/prepare/land/3/1975/3-1975-daily-CDM_lite_SecondRelease_AF000040.psv
[INFO] Wrote success file: /gws/smf/j04/c3s311a_lot2/workflow/r2.0/lite/land/outputs/log/success/3/3-1975-daily-CDM_lite_SecondRelease_AF000040.psv
Test the prepare all script:
./scripts/land/prepare-all-land-lite.sh r2.0 local
It should keep running, but it is running locally, so eventually we want to exit it (because it will take forever if not run on LOTUS).
When re-running all, we need to delete success and failure files, e.g.:
find /gws/smf/j04/c3s311a_lot2/workflow/r2.0/lite/land/outputs/log/ -type f -exec rm {} \;
I refactored this into a local script, based on my own paths:
$ more clear_log_data_dirs_prepare_land_cdmlite.sh
find /work/scratch-nopw/astephen/glamod/r2.0/cdmlite/prepare /gws/smf/j04/c3s311a_lot2/workflow/r2.0/lite/land/outputs/log -type f -exec rm {} \;
rm -f last-land-batch.txt
This might take some days, depending on the LOTUS load.
./scripts/land/prepare-all-land-lite.sh r2.0 batch
This script limits the number of jobs in the queue to 250, and keeps sleeping then submitting more.
It also caches the last batch ID that was run to a local file (last-land-batch.txt
) and when you re-run it, it will wait for a match of that before re-running/re-submitting any batches.
Indeed, essentially, you need to re-run all and check that there are no errors in the LOTUS outputs.
Here are some of the things I did:
A script to manage the whole task in about ~150 jobs on LOTUS:
./scripts/helpers/wrap-restructure-land-big-batches.sh
Which uses:
./scripts/helpers/restructure-land-batches-in-100s.sh
The LOTUS output files are all then written to the top-level directory. You can examine the sizes of them to check if they all ran.
./scripts/land/create-sqls-land.sh r2.0 0
./scripts/land/create-sqls-land.sh r2.0 2
./scripts/land/create-sqls-land.sh r2.0 3
./scripts/land/load-land-sql.sh r2.0 0
./scripts/land/load-land-sql.sh r2.0 2
./scripts/land/load-land-sql.sh r2.0 3
You can check for any errors in the logs by gripping everything other than "COPY \d+". E.g.:
$ grep -vP "COPY a\d+" /gws/smf/j04/c3s311a_lot2/workflow/r2.0/lite/land/populate/log/load-0-*.sql.log
$ grep -vP "COPY a\d+" /gws/smf/j04/c3s311a_lot2/workflow/r2.0/lite/land/populate/log/load-2-*.sql.log
$ grep -vP "COPY a\d+" /gws/smf/j04/c3s311a_lot2/workflow/r2.0/lite/land/populate/log/load-3-*.sql.log
If there is no output then we assume it worked!
You might find that the load
scripts are not able to connect to the database on the production server (glamod2.ceda.ac.uk
). To fix this, update the iptables
and pg_hba.conf
files on the production server so that LOTUS nodes and/or sci
servers can connect to the server.
Land processing
Login and set up environment
Now decide the batch sizes to fix upon - for land data
For a new release
The output should look like: