search

GeoscienceAustralia / dea-cogger

Convert NetCDFs to Cloud Optimized GeoTIFFs
15 stars 4 forks source link

Cog auto ashoka #14

Closed ashoka1234 closed 5 years ago

ashoka1234 commented 5 years ago

Request for this pull request

Work list generation for products

Proposed solution

  1. Config file aws_products_config.yaml defines the prefix and name_template for each product. The prefix is a top-level fixed part of the product file keys while name_template defines variable part to be used for each dataset. File extension and band names are to be appended to the combination of prefix and name_template in order to compute file names corresponding to datasets.
  2. Work list generation script work_list.py does the following: 1) Extract datasets from datacube: It uses product name_template to identify field_names required for datacube query. 2) Construct/fill the corresponding name_template using the information supplied by the datacube query and generate a list of expected .yaml files. 3) Filter AWS inventory list to extract existing .yaml files corresponding to datasets for the given product. 4) Subtract (3) from (2) and write the list of files to file_list in output_dir specified
ashoka1234 commented 5 years ago

That was a nice one liner re.findall. Thanks Cheers UNOFFICIAL Classified by ashoka.jayawardena@ga.gov.au on 19/12/2018 11:52:42 AM From: Damien Ayers notifications@github.com Sent: Wednesday, 19 December 2018 11:39 AM To: GeoscienceAustralia/COG-Conversion COG-Conversion@noreply.github.com Cc: Jayawardena Ashoka Ashoka.Jayawardena@ga.gov.au; Author author@noreply.github.com Subject: Re: [GeoscienceAustralia/COG-Conversion] Cog auto ashoka (#14)

@omad approved this pull request.

Looks great Ashoka.

I've made a few little tidy ups, see my last commit for details.

Let me know if you'd like to discuss anything, or if you'd prefer that I left PR review comments instead of just changing the code!

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/GeoscienceAustralia/COG-Conversion/pull/14#pullrequestreview-186345846, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AHhwa7rWL-VqJqvscL4sWVqpeEK9fktFks5u6YqUgaJpZM4ZXj5T.

Geoscience Australia Disclaimer: This e-mail (and files transmitted with it) is intended only for the person or entity to which it is addressed. If you are not the intended recipient, then you have received this e-mail by mistake and any use, dissemination, forwarding, printing or copying of this e-mail and its file attachments is prohibited. The security of emails transmitted cannot be guaranteed; by forwarding or replying to this email, you acknowledge and accept these risks.