search

glamod / glamod-ingest

Database preparation and ingestion for GLAMOD
BSD 2-Clause "Simplified" License
2 stars 1 forks source link

Load marine data into production DB #53

Closed agstephens closed 3 years ago

agstephens commented 3 years ago

Marine processing pipeline

  1. Restructure
  2. Generate SQL scripts
  3. Load the DB

1. Restructure

Test locally:

./scripts/marine/prepare-all-marine-lite.sh r2.0 local

When working, run all in batch mode:

./scripts/marine/prepare-all-marine-lite.sh r2.0 batch

When all jobs have completed, re-run after running on LOTUS - to pick up anything in logs that is not expected:

nohup ./scripts/marine/prepare-all-marine-lite.sh r2.0 batch 2>&1 > prepare.marine.output.txt &

Look for "error" and "exception" in the output file, and anything not matching "INFO", "TIMER" etc.

2. Generate SQL scripts

When all the above have run okay...

./scripts/marine/create-sqls-marine.sh 0 r2.0

This generates all the loader scripts in...

/work/scratch-nopw/astephen/glamod/r2.0/cdmlite/prepare/marine/<YEAR>/*.sql

3. Load the DB

Now load the tables into the DB:

nohup ./scripts/marine/load-marine-sql.sh 0 > load.marine.0.txt &