dlr-eoc / prosEO

prosEO – A Processing System for Earth Observation Data
GNU General Public License v3.0
14 stars 1 forks source link

Multiple input files of same file type in Job Order file create separate input elements #67

Closed tangobravo62 closed 3 years ago

tangobravo62 commented 4 years ago

The Generic IPF Interface Specification does not state, whether it is allowed to have multiple input elements with the same file type in a Job Order file. However, it is known that some processors do not accept multiple entries with the same file type (e. g. S5P KNMI L2 processor rejects multiple DEM configuration file entries, S5P DLR L2 BGCAL processor expects all input files of the same file type as a list of file names in a single entry).

Currently if the input selection finds multiple products satisfying a product query, multiple input entries are generated, each with the product ID of the respective product. Each input entry may have multiple file names, if the product itself consists of several (data and auxiliary) files.

It has to be decided whether we keep this design and offload the task to unify the input file lists to the processor-specific wrapper, or whether the Production Planner already collects all files of the same file type into a single entry. This would entail an interface change between Planner and Base Wrapper, since the product ID currently is an attribute of the "Input" element, not of the "File_Name" element (which it then would have to be).

tangobravo62 commented 4 years ago

Discussion 2020-06-22: Since the product ID of an input element is not used by the Base Wrapper, it can safely be dropped (it is still needed on the output element, of which by basic prosEO design assumption there can only be one per processor run anyway, so we would not expect problems here).

The decision therefore is to collect all output files of the same product class in a single input element.

emelchinger commented 4 years ago

Implemented

tangobravo62 commented 3 years ago

Successfully tested for various mission configurations.