[BUG] future ignoreFilesBefore date is replace by deployment date

[Woljtek]

Environment:

• Delivery tag: 1.5
• Platform: OPS Orange Cloud
• Configuration: production-common: 1.13

Traçability:

• Test(s): Test 24h

Current Behavior: During the preparation of the 24h test, we redeployed the ingestion chains with a ignoreFilesBefore date in the future. The ignoreFilesBefore date is replaced by the deployment date. So we started the ingestion earlier than expected.

Expected Behavior: The configured ignoreFilesBefore date in stream.parameters.properties shall be used.

Steps To Reproduce: 1- Set a ignoreFilesBefore in the future 2- Deploy the ingestion chain 3- Obverve that chunks are ingested after the deployment date

Test execution artifacts (i.e. logs, screenshots…) Example with ingestion-pedc:

1. Deployment:
2. Logs in trigger:

   {"header":{"type":"LOG","timestamp":"2023-04-27T13:05:01.718744Z","level":"INFO","line":55,"file":"StartupInfoLogger.java","thread":"main"},"message":{"content":"Starting Application using Java 11.0.18 on ingestion-pedc-part1-ingestion-edip-trigger-v79-7bb67bd96-qt8nc with PID 1 (/app/rs-ingestion-trigger.jar started by rsuser in /app)"},"custom":{"logger_string":"esa.s1pdgs.cpoc.ingestion.trigger.Application"}}
   {"header":{"type":"LOG","timestamp":"2023-04-27T13:05:01.727440Z","level":"INFO","line":640,"file":"SpringApplication.java","thread":"main"},"message":{"content":"No active profile set, falling back to 1 default profile: \"default\""},"custom":{"logger_string":"esa.s1pdgs.cpoc.ingestion.trigger.Application"}}
   {"header":{"type":"LOG","timestamp":"2023-04-27T13:05:10.008147Z","level":"INFO","line":30,"file":"MongoConfiguration.java","thread":"main"},"message":{"content":"Create new mongo client"},"custom":{"logger_string":"esa.s1pdgs.cpoc.ingestion.trigger.config.MongoConfiguration"}}
   {"header":{"type":"LOG","timestamp":"2023-04-27T13:05:11.918265Z","level":"WARN","line":217,"file":"JavaBeanExecutable.java","thread":"main"},"message":{"content":"HV000254: Missing parameter metadata for ProductFamily(String, int), which declares implicit or synthetic parameters. Automatic resolution of generic type information for method parameters may yield incorrect results if multiple parameters have the same erasure. To solve this, compile your code with the '-parameters' flag."},"custom":{"logger_string":"org.hibernate.validator.internal.properties.javabean.JavaBeanExecutable"}}
   {"header":{"type":"LOG","timestamp":"2023-04-27T13:05:12.298998Z","level":"INFO","line":41,"file":"IngestionTriggerServiceConfiguration.java","thread":"main"},"message":{"content":"Adding new inbox to be polled: Inbox [inboxAdapter=EdipInboxAdapter [inboxDirectory=ftps://154.14.100.130:21/NOMINAL], filter=JoinedFilter [filters=[BlacklistRegexRelativePathInboxFilter [pattern=$a], WhitelistRegexRelativePathInboxFilter [pattern=^([A-Za-z_]{4}/)?([0-9A-Za-z_]{1})2([0-9A-Za-z_]{1})/([0-9A-Za-z_]+)/(ch[0|_]?[1-2]/)?(DCS_[0-9]{2}_([a-zA-Z0-9_]*)_ch([12])_(DSDB|DSIB).*\\.(raw|aisp|xml|RAW|AISP|XML))$], MinimumModificationDateFilter [ignoreFilesBefore=Mon Jan 23 23:59:59 UTC 2023]]]]"},"custom":{"logger_string":"esa.s1pdgs.cpoc.ingestion.trigger.config.IngestionTriggerServiceConfiguration"}}
   {"header":{"type":"LOG","timestamp":"2023-04-27T13:05:12.301336Z","level":"INFO","line":41,"file":"IngestionTriggerServiceConfiguration.java","thread":"main"},"message":{"content":"Adding new inbox to be polled: Inbox [inboxAdapter=EdipInboxAdapter [inboxDirectory=ftps://154.14.100.130:21/NOMINAL], filter=JoinedFilter [filters=[BlacklistRegexRelativePathInboxFilter [pattern=$a], WhitelistRegexRelativePathInboxFilter [pattern=^([A-Za-z_]{4}/)?([0-9A-Za-z_]{1})1([0-9A-Za-z_]{1})/([0-9A-Za-z_]+)/(ch[0|_]?[1-2]/)?(DCS_[0-9]{2}_([a-zA-Z0-9_]*)_ch([12])_(DSDB|DSIB).*\\.(raw|aisp|xml|RAW|AISP|XML))$], MinimumModificationDateFilter [ignoreFilesBefore=Thu Apr 27 13:05:12 UTC 2023]]]]"},"custom":{"logger_string":"esa.s1pdgs.cpoc.ingestion.trigger.config.IngestionTriggerServiceConfiguration"}}
   {"header":{"type":"LOG","timestamp":"2023-04-27T13:05:12.308465Z","level":"INFO","line":64,"file":"KafkaProducerConfiguration.java","thread":"main"},"message":{"content":"using producer config {retries=0, value.serializer=class org.springframework.kafka.support.serializer.JsonSerializer, bootstrap.servers=, spring.json.add.type.headers=false, key.serializer=class org.apache.kafka.common.serialization.StringSerializer}"},"custom":{"logger_string":"esa.s1pdgs.cpoc.message.kafka.config.KafkaProducerConfiguration"}}
   {"header":{"type":"LOG","timestamp":"2023-04-27T13:05:22.805226Z","level":"INFO","line":61,"file":"StartupInfoLogger.java","thread":"main"},"message":{"content":"Started Application in 23.506 seconds (JVM running for 26.404)"},"custom":{"logger_string":"esa.s1pdgs.cpoc.ingestion.trigger.Application"}}
   {"header":{"type":"LOG","timestamp":"2023-04-27T13:06:34.490320Z","level":"INFO","line":162,"file":"PollingRun.java","thread":"scheduling-1"},"message":{"content":"Handled 178 finished elements"},"custom":{"logger_string":"esa.s1pdgs.cpoc.ingestion.trigger.inbox.Inbox (WILE) for EDRS_SESSION"}}
   {"header":{"type":"LOG","timestamp":"2023-04-27T13:07:41.252358Z","level":"INFO","line":162,"file":"PollingRun.java","thread":"scheduling-1"},"message":{"content":"Handled 178 finished elements"},"custom":{"logger_string":"esa.s1pdgs.cpoc.ingestion.trigger.inbox.Inbox (WILE) for EDRS_SESSION"}}
   {"header":{"type":"REPORT","timestamp":"2023-04-27T13:34:47.194000Z","level":"INFO","mission":"S2","workflow":"NOMINAL","rs_chain_name":"INGESTION-PEDC","rs_chain_version":"1.12.0-rc1"},"message":{"content":"New file detected S1A/DCS_02_L20230322101944355000076_dat/ch_1/DCS_02_L20230322101944355000076_ch1_DSDB_00001.raw"},"task":{"uid":"837037b6-5766-467b-99a7-c5f1c65e3ba1","name":"IngestionTrigger","event":"BEGIN","input":{"pickup_point_detection_date":"2023-04-27T13:34:47.192000Z","pickup_point_available_date":"2023-04-27T13:28:00.000000Z","filename_string":"DCS_02_L20230322101944355000076_ch1_DSDB_00001.raw"}}}
   {"header":{"type":"REPORT","timestamp":"2023-04-27T13:34:47.208000Z","level":"INFO","mission":"S2","workflow":"NOMINAL","rs_chain_name":"INGESTION-PEDC","rs_chain_version":"1.12.0-rc1"},"message":{"content":"File S1A/DCS_02_L20230322101944355000076_dat/ch_1/DCS_02_L20230322101944355000076_ch1_DSDB_00001.raw is ignored by filter."},"task":{"uid":"837037b6-5766-467b-99a7-c5f1c65e3ba1","name":"IngestionTrigger","event":"END","status":"OK","output":{},"input":{"pickup_point_detection_date":"2023-04-27T13:34:47.192000Z","pickup_point_available_date":"2023-04-27T13:28:00.000000Z","filename_string":"DCS_02_L20230322101944355000076_ch1_DSDB_00001.raw"},"quality":{},"error_code":0,"duration_in_seconds":0.014,"missing_output":[]}}
   {"header":{"type":"REPORT","timestamp":"2023-04-27T13:34:47.295000Z","level":"INFO","mission":"S2","workflow":"NOMINAL","rs_chain_name":"INGESTION-PEDC","rs_chain_version":"1.12.0-rc1"},"message":{"content":"New file detected S1A/DCS_02_L20230322101944355000076_dat/ch_1/DCS_02_L20230322101944355000076_ch1_DSDB_00002.raw"},"task":{"uid":"a213efeb-bf9d-40b3-ae84-b0d430b3f13d","name":"IngestionTrigger","event":"BEGIN","input":{"pickup_point_detection_date":"2023-04-27T13:34:47.295000Z","pickup_point_available_date":"2023-04-27T13:28:00.000000Z","filename_string":"DCS_02_L20230322101944355000076_ch1_DSDB_00002.raw"}}}

   => MinimumModificationDateFilter [ignoreFilesBefore=Thu Apr 27 13:05:12 UTC 2023]

3- Deployed configuration:

Whenever possible, first analysis of the root cause

Bug Generic Definition of Ready (DoR)

• [x] The affect version in which the bug has been found is mentioned
• [x] The context and environment of the bug is detailed
• [x] The description of the bug is clear and unambiguous
• [x] The procedure (steps) to reproduce the bug is clearly detailed
• [ ] The tested User Story / features is linked to the bug if available
• [x] Logs are attached if available
• [ ] A data set attached if available

Bug Generic Definition of Done (DoD)

• [ ] the modification implemented (the solution to fix the bug) is described in the bug.
• [ ] Unit tests & Continuous integration performed - Test results available - Structural Test coverage reported by SONAR
• [ ] Code committed in GIT with right tag or Analysis/Trade Off documentation up-to-date in reference-system-documentation repository
• [ ] Code is compliant with coding rules (SONAR Report as evidence)
• [ ] Acceptance criteria of the related User story are checked and Passed

[Woljtek]

Because the ingestion redeployments are rare, the priority is minor.

[w-jka]

This behaviour is not a bug, but indeed wanted behaviour. Unfortunately, the JIRA from the S1PRO development is not active anymore, so I am unable to refer to the original request as to why this was implemented (If you have access to it still the issue number was S1PRO-2098).

[Woljtek]

The current behavior is not described in the inboxes section10. Is there a place where this mechanism inherited from S1PRO is described?

[w-fsi]

The behaviour might be described in the story of the old jira system. As it is down, it is difficult to trace the intensions. The idea behind the ignoreFilter is to avoid that the whole data of a server is transferred when a fresh installation is setup. The intension was not to allow a timing of when the ingestion trigger start to pull in files.

My assumption is that there was a request to set the data to know, if it its in the future as someone was confused why the system is not pulling in data. This behaviour was existing since the S1PRO and thus would be technically a change of behaviour. The code is clearily indicating that the current behaviour was wanted:

        // S1PRO-2098: If the configured date is in the future, take the current date as the
        // reference value
        if (ignoreFilesBeforeDate.after(now)) {
            return now;
        }

Removing this behaviour would be easy however.

[LAQU156]

IVV_CCB_2023_w18 : Moved into "Accepted Werum" to be discussed if documentation or code should be changed, Priority minor, no need for phase 1

[LAQU156]

Werum_CCB_2023_w18 : Moved into "Product Backlog", documentation issue

[w-fsi]

Documentation was updated to reflect that a future data will be set to the current time.

[w-fsi]

This issue was tackled in V1.13.1 and thus will be contained in V2. (refer to https://github.com/COPRS/production-common/releases/tag/1.13.1-rc1)

[pcuq-ads]

SYS_CCB_w29 : A comment has been added on the default configuration from release 1.13.1.