search

GeoscienceAustralia / dea-orchestration

4 stars 1 forks source link

Automate s2nbar rolling archive #141

Closed santoshamohan closed 4 years ago

santoshamohan commented 4 years ago

Background

The Sentinel-2 Definitive NBAR product is required to be a rolling archive stored on AWS S3 for the last two years. This process requires serverless automation to kick start weekly generation of new datasets once S2 ARD Granules are successfully generated by execute_ssh_command_js serverless lambda function.

Required Manual steps

Once the ARD process is completed, we need to run the wagl batch_summary command to generate list of successfully processed s2 level1 datasets. This list is a text file, with each line being a level1 S2 Zip file, that has been processed through to ARD.

It is then necessary to update the LEVEL1_DONE_LIST in the automate_s2_nbar_rolling_archive/serverless.yml file, to point to the above text file.

Following are the updates within this PR

How the process works.

The Lambda executes the raijin_scripts/execute_s2nbar_rolling_archive/run script on raijin. This script simply submits a PBS job which executes submit_s2_copyq_jobs.sh.

submit_s2_copyq runs on a copyq node to access S3, and:

  1. Reads from the list of level1 datasets that have been successfully processed.
  2. Turns each level 1 file name into the path to an ARD dataset.
  3. Uploads everything except NBAR-T.
  4. Uploads the dataset document
    • Removes lineage
    • Removes NBAR-T measurements
    • Generate a new Dataset ID
codecov[bot] commented 4 years ago

Codecov Report

Merging #141 into master will increase coverage by 2.21%. The diff coverage is 97.11%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #141      +/-   ##
==========================================
+ Coverage   55.84%   58.05%   +2.21%     
==========================================
  Files          28       32       +4     
  Lines        1832     1936     +104     
==========================================
+ Hits         1023     1124     +101     
- Misses        809      812       +3
Impacted Files Coverage Δ
...archive/run_s3_update_sync/test_update_metadata.py 100% <100%> (ø)
...bar_rolling_archive/execute_s2nbar/test_handler.py 100% <100%> (ø)
..._s2_nbar_rolling_archive/execute_s2nbar/handler.py 88.88% <88.88%> (ø)
...ling_archive/run_s3_update_sync/update_metadata.py 97.05% <97.05%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 17ca80d...8c127a3. Read the comment docs.

santoshamohan commented 4 years ago

Thanks for the comments, Damien. I will incorporate your comments and request re-review.

omad commented 4 years ago

Superseded by #166