First steps in building a prototype. The modis-ingestor is the main repository of code that is run on a daily basis. The ingestor should be able to determine the last data available on S3, query and locate new data since that point (and missing/updated data?), fetch the data, split into individual datasets, and uploaded to S3 following standard directory structure of:
PRODUCTNAME/TILEID/DATE
where PRODUCTNAME is a product shortname and version ID (e.g., MCD43A4.006) and TILEID can either be a single directory or multiple directories that make up the tile identifier. For example Landsat uses 006/109 (path/row) but MODIS may use h05v07, or 05/07. DATE is either a directory of the date or at least a scene ID that is unique within that TILEID for a specific date.
AWS Lambda should be sufficient to do all required processing and uploading to S3 within the time limits imposed by Lambda.
drewbo
commented
7 years ago
First steps in building a prototype. The modis-ingestor is the main repository of code that is run on a daily basis. The ingestor should be able to determine the last data available on S3, query and locate new data since that point (and missing/updated data?), fetch the data, split into individual datasets, and uploaded to S3 following standard directory structure of: PRODUCTNAME/TILEID/DATE
where PRODUCTNAME is a product shortname and version ID (e.g., MCD43A4.006) and TILEID can either be a single directory or multiple directories that make up the tile identifier. For example Landsat uses 006/109 (path/row) but MODIS may use h05v07, or 05/07. DATE is either a directory of the date or at least a scene ID that is unique within that TILEID for a specific date.
AWS Lambda should be sufficient to do all required processing and uploading to S3 within the time limits imposed by Lambda.