search

glamod / glamod-ingest

Database preparation and ingestion for GLAMOD
BSD 2-Clause "Simplified" License
2 stars 1 forks source link

Daily updates - land data #78

Open agstephens opened 3 years ago

agstephens commented 3 years ago

@jhaigh0 As discussed with Simon, here is our plan for running daily updates:

  1. Outline:

    • we want a single script that will:
    • find the latest cdmlite daily update files
    • attempt to process each file:
      1. Restructure file to PSV under: /work/scratch...
      2. Generate SQL command to load the new file
      3. Execute the SQL command
      4. Remove the input file(s) under: /work/scratch...

  2. Data locations and proposed workflow:

    • data will be found in:
    • /gws/nopw/j04/c3s311a_lot2/data/level2/land/r202102/daily_updates/cdm_lite/
    • In the subdirectory incoming/
    • for each file we process, we will:
      1. Move it to processing/
      2. Attempt to process it (as outlined above)
      3. If any part of the process fails, move it failed/ and log/email result
      4. If it succeeds, move it to complete/ and log/email result

  3. Script details:

    • the script can be bash or python
    • it should be called something like: ./scripts/land/run-daily-update-cdmlite.(py|sh)
    • it will need to use the settings file to get the location of the .../daily_updates/cdm_lite/ directory for a given release.

  4. Once it is all working:

    • we will schedule it to run once per day - probably submitted to a LOTUS node, but kicked off by the cron server.