Open-EO / openeo-geopyspark-driver

OpenEO driver for GeoPySpark (Geotrellis)
Apache License 2.0
26 stars 4 forks source link

Save_asset_per_band: assets in different s3 folders + band-specific filename #877

Open JorisCod opened 1 week ago

JorisCod commented 1 week ago

The goal of this issue is to save assets to different folders, each with a specific name according to the band name.

Specific use case in mind: saving of monthly composites in different s3 folders (one folder/month)

Prefix example: vito/products/2020/'<MONTH'>/

Filename example: LCFMLSF-MONTHLY__29TNE_V001-SATIO_B_M.tif

So what's parametric is month, band_name, resolution, date_start and date_end. Resolution can be considered fixed (*). Month and band_name can be derived as follows:

band_name = openeo_bandname.split('')[-1] month = openeo_bandname.split('')[0]

(alternatively with fixed indices).

date_start and date_end are harder as they are to be derived from month:

import calendar
from datetime import datetime

def get_start_and_end_dates(year: int, month: int):
    # Get the first day of the month
    start_date = datetime(year, month, 1)

    # Get the last day of the month
    last_day = calendar.monthrange(year, month)[1]
    end_date = datetime(year, month, last_day)

    return start_date, end_date

# Example usage
year = 2020
month = 2
start_date, end_date = get_start_and_end_dates(year, month)

print("Start date:", start_date.strftime('%Y%m%d'))
print("End date:", end_date.strftime('%Y%m%d'))

The solution might be to provide a list of prefixes and filenames to use, both lists having the same length as the number of bands. This is quite general as well as flexible.

'* An ideal workflow is to have 1 job for all resolutions. This is actually the goal of main workflow of LCFM, to have a workflow for both 10m and 20m bands.

jdries commented 1 week ago

@JeroenVerstraelen to plan in next sprint a list or dictionary mapping band names to preferred asset name is indeed a way forward. Allowing subdirectory in prefix is (hopefully) also possible.

bossie commented 1 day ago

Related to https://github.com/Open-EO/openeo-geotrellis-extensions/issues/317 in that this also accepts a dictionary mapping band names to scale, offset etc.