search

dlr-eoc / prosEO

prosEO – A Processing System for Earth Observation Data
GNU General Public License v3.0
14 stars 1 forks source link

Set filters per input type and parameter per output type in processing order #81

Closed tangobravo62 closed 3 years ago

tangobravo62 commented 4 years ago

Filter conditions cannot be applied selectively to different input product classes: Below, the input filter "tile:26PNB" is applied first to L1C_TL (correct, succeeds) and then to L2A_DS (wrong, fails):

2020-06-27 05:42:44.223  INFO 1 --- [nio-8080-exec-2] de.dlr.proseo.planner.util.OrderUtil     : (I3010) Processing order 'L2A_TL TDS1_ORBIT_15250-00002' is released
2020-06-27 05:42:44.226 TRACE 1 --- [nio-8080-exec-2] de.dlr.proseo.planner.util.JobStepUtil   : Looking for product queries of job step: 2972
2020-06-27 05:42:44.239 TRACE 1 --- [nio-8080-exec-2] d.d.p.model.service.ProductQueryService  : >>> executeQuery(useNativeSQL = false)
2020-06-27 05:42:44.271 TRACE 1 --- [nio-8080-exec-2] d.d.p.model.service.ProductQueryService  : Number of products in database: 239
2020-06-27 05:42:44.272 DEBUG 1 --- [nio-8080-exec-2] d.d.p.model.service.ProductQueryService  : Executing JPQL query: select p from Product p where p.productClass.id = 118 and p.sensingStartTime <= '2020-02-06 13:51:04.000000' and p.sensingStopTime >= '2020-02-06 13:51:04.000000' and p.generationTime >= all (select p2.generationTime from Product p2 where p2.productClass.id = 118 and p2.sensingStartTime <= '2020-02-06 13:51:04.000000' and p2.sensingStopTime >= '2020-02-06 13:51:04.000000')
2020-06-27 05:42:44.287 TRACE 1 --- [nio-8080-exec-2] d.d.p.model.service.ProductQueryService  : Number of products found: 23
2020-06-27 05:42:44.290 TRACE 1 --- [nio-8080-exec-2] d.d.p.model.service.ProductQueryService  : Found product 2520 with start time = 2020-02-06T13:51:04Z, stop time = 2020-02-06T13:51:04Z and revision = 
[...] 
2020-06-27 05:42:44.305 TRACE 1 --- [nio-8080-exec-2] d.d.p.model.service.ProductQueryService  : Found product 2564 with start time = 2020-02-06T13:51:04Z, stop time = 2020-02-06T13:51:04Z and revision = 
2020-06-27 05:42:44.402 TRACE 1 --- [nio-8080-exec-2] d.d.p.model.service.ProductQueryService  : Number of products after selection: 1
2020-06-27 05:42:44.404 TRACE 1 --- [nio-8080-exec-2] d.d.p.model.service.ProductQueryService  : Number of products after testing filter conditions: 1
2020-06-27 05:42:44.426 TRACE 1 --- [nio-8080-exec-2] d.d.p.model.service.ProductQueryService  : <<< executeQuery()
2020-06-27 05:42:44.447 TRACE 1 --- [nio-8080-exec-2] d.d.p.model.service.ProductQueryService  : >>> executeQuery(useNativeSQL = false)
2020-06-27 05:42:44.455 TRACE 1 --- [nio-8080-exec-2] d.d.p.model.service.ProductQueryService  : Number of products in database: 239
2020-06-27 05:42:44.455 DEBUG 1 --- [nio-8080-exec-2] d.d.p.model.service.ProductQueryService  : Executing JPQL query: select p from Product p where p.productClass.id = 124 and p.sensingStartTime <= '2020-02-06 13:51:04.000000' and p.sensingStopTime >= '2020-02-06 13:51:04.000000' and p.generationTime >= all (select p2.generationTime from Product p2 where p2.productClass.id = 124 and p2.sensingStartTime <= '2020-02-06 13:51:04.000000' and p2.sensingStopTime >= '2020-02-06 13:51:04.000000')
2020-06-27 05:42:44.502 TRACE 1 --- [nio-8080-exec-2] d.d.p.model.service.ProductQueryService  : Number of products found: 1
2020-06-27 05:42:44.503 TRACE 1 --- [nio-8080-exec-2] d.d.p.model.service.ProductQueryService  : Found product 2952 with start time = 2020-02-06T13:51:04Z, stop time = 2020-02-06T13:51:04Z and revision = 
2020-06-27 05:42:44.508 TRACE 1 --- [nio-8080-exec-2] d.d.p.model.service.ProductQueryService  : Number of products after selection: 1
2020-06-27 05:42:44.560 TRACE 1 --- [nio-8080-exec-2] d.d.p.model.service.ProductQueryService  : Product [productClass=L2A_DS, mode=generate_datastrip, sensingStartTime=2020-02-06T13:51:04Z, sensingStopTime=2020-02-06T13:51:04Z, generationTime=2020-06-26T17:16:50.958399Z, productFile=[ProductFile [processingFacility=ProcessingFacility [name=telekom-otc, description=Telekom_OTC, processingEngineUrl=https://proseo-k8s-gate.de/, storageManagerUrl=https://proseo-k8s-gate.de/api/v1/namespaces/default/services/storage-mgr-service:service/proxy/proseo/storage-mgr/v1/], productFileName=S2B_OPER_MSI_L2A_DS_DLR__20200626T171658_S20200206T121645_N02.14.tar, auxFileNames=[], filePath=s3://proseo-data-001/2952, storageType=S3]], parameters={}] does not meet filter conditions
2020-06-27 05:42:44.560 TRACE 1 --- [nio-8080-exec-2] d.d.p.model.service.ProductQueryService  : <<< executeQuery()
2020-06-27 05:42:44.568 TRACE 1 --- [nio-8080-exec-2] de.dlr.proseo.model.util.OrderUtil       : >>> toRestOrder(2960)

It must be possible to apply filter conditions to a single input product class in the processing order. (This is not only true for the example above, but also when selecting auxiliary products, which usually do not have the same parameters attached as the mission products.)

tangobravo62 commented 4 years ago

The same applies to the output parameters: These may be different per output product type as well!

tangobravo62 commented 4 years ago

order_change.json.txt Order input filters per product class can now be set and changed (first entry shows DUMMY product class after database update):

prosEO (PTM)> order show
---
- id: 80
  version: 3
  missionCode: "PTM"
  identifier: "L3_products_9:30-17:30"
  [...]
  inputFilters:
  - productClass: "DUMMY"
    filterConditions:
    - key: "fileClass"
      parameterType: "STRING"
      parameterValue: "TEST"
    - key: "revision"
      parameterType: "INTEGER"
      parameterValue: "99"
  classOutputParameters: []
  [...]

prosEO (PTM)> order update --file=order_change.json
(I2938) Order with identifier L3_products_9:30-17:30 updated (new version 4)

prosEO (PTM)> order show L3_products_9:30-17:30
---
- id: 80
  version: 4
  missionCode: "PTM"
  identifier: "L3_products_9:30-17:30"
  [...]
  inputFilters:
  - productClass: "L1B_______"
    filterConditions:
    - key: "fileClass"
      parameterType: "STRING"
      parameterValue: "TEST"
    - key: "revision"
      parameterType: "INTEGER"
      parameterValue: "99"
  classOutputParameters: []
  [...]

Class-specific output parameters can be set as well (from attached file order_change.json):

prosEO (PTM)> order update --file=order_change.json
(I2938) Order with identifier L3_products_9:30-17:30 updated (new version 5)
prosEO (PTM)> order show L3_products_9:30-17:30
---
- id: 80
  version: 5
  missionCode: "PTM"
  identifier: "L3_products_9:30-17:30"
  [...]
  classOutputParameters:
  - productClass: "PTM_L3"
    outputParameters:
    - key: "copernicusCollection"
      parameterType: "INTEGER"
      parameterValue: "13"
    - key: "revision"
      parameterType: "INTEGER"
      parameterValue: "7"
  outputParameters:
  - key: "copernicusCollection"
    parameterType: "INTEGER"
    parameterValue: "77"
  - key: "revision"
    parameterType: "INTEGER"
    parameterValue: "99"
  [...]

Upon product generation the class-specific parameters for PTM_L3 now should supersede the general parameters --> to be tested.

tangobravo62 commented 3 years ago

Tested with order L2_orbits_3000-3002 in mission PTM: Output product of type L1B_PART1 correctly shows revision 2 instead of revision 99 (global output parameter).