Closed agstephens closed 3 years ago
Test locally:
./scripts/marine/prepare-all-marine-lite.sh r2.0 local
When working, run all in batch mode:
./scripts/marine/prepare-all-marine-lite.sh r2.0 batch
When all jobs have completed, re-run after running on LOTUS - to pick up anything in logs that is not expected:
nohup ./scripts/marine/prepare-all-marine-lite.sh r2.0 batch 2>&1 > prepare.marine.output.txt &
Look for "error" and "exception" in the output file, and anything not matching "INFO", "TIMER" etc.
When all the above have run okay...
./scripts/marine/create-sqls-marine.sh 0 r2.0
This generates all the loader scripts in...
/work/scratch-nopw/astephen/glamod/r2.0/cdmlite/prepare/marine/<YEAR>/*.sql
Now load the tables into the DB:
nohup ./scripts/marine/load-marine-sql.sh 0 > load.marine.0.txt &
Marine processing pipeline
1. Restructure
Test locally:
When working, run all in batch mode:
When all jobs have completed, re-run after running on LOTUS - to pick up anything in logs that is not expected:
Look for "error" and "exception" in the output file, and anything not matching "INFO", "TIMER" etc.
2. Generate SQL scripts
When all the above have run okay...
This generates all the loader scripts in...
3. Load the DB
Now load the tables into the DB: