[BUG] S3-PUG-STC IPF crashes on some S3A_SY_2_SYN products

eroan-marie commented 1 year ago

Environment:

Delivery tag:1.13.2-rc1
Platform: OPS Orange Cloud
Configuration:

Traceability:

Current Behavior:

IPF crashes while processing some S3A_SY_2_SYN products

Expected Behavior:

Processing should work fine

Steps To Reproduce:

Try processing several S3A_SY_2_SYN products and see error below

Test execution artefacts (i.e. logs, screenshots…) Tip: You can attach images or log files by dragging & dropping, selecting or pasting them. s3-pug-stc.log

Whenever possible, first analysis of the root cause

Bug Generic Definition of Ready (DoR)

[X] The affect version in which the bug has been found is mentioned
[X] The context and environment of the bug is detailed
[X] The description of the bug is clear and unambiguous
[X] The procedure (steps) to reproduce the bug is clearly detailed
[ ] The tested User Story / features is linked to the bug if available
[X] Logs are attached if available
[ ] A data set attached if available

Bug Generic Definition of Done (DoD)

[ ] the modification implemented (the solution to fix the bug) is described in the bug.
[ ] Unit tests & Continuous integration performed - Test results available - Structural Test coverage reported by SONAR
[ ] Code committed in GIT with right tag or Analysis/Trade Off documentation up-to-date in reference-system-documentation repository
[ ] Code is compliant with coding rules (SONAR Report as evidence)
[ ] Acceptance criteria of the related User story are checked and Passed

w-jka commented 1 year ago

@eroan-marie Based on the provided logs, it seems, that two identical SYN intervals with different creation times were provided to the IPF. I would assume, that this might be the root cause, why the processor ran into an error in the NetCDF library.

LAQU156 commented 1 year ago

IVV_CCB_2023_w28 : Moved into "Accepted Werum" for correction, Priority blocking, seems similar as #1022, to be fixed phase 1

LAQU156 commented 1 year ago

Werum_CCB_2023_w28 : Moved into "Product Backlog" to see if a fix can be provided without change the tasktable

w-jka commented 1 year ago

@eroan-marie After further investigation this issue does not seem to be related to #1022. There are no duplicated input validation time frames. As the issue happens inside of the IPF logic, and is related to the NetCDF library, this issue is not fixable by Werum in its current state. Please refer to the IPF provider with the logs of the IPF.

Candidate for refusal.

pcuq-ads commented 1 year ago

Werum_CCB_2023_w30 : One hypothesis is that this could be a baseline issue. This is under test on ADS side.

SYTHIER-ADS commented 1 year ago

After the test with a consistent baseline, we have exactly the same behavior. Issue is to be analysed by Werum

w-jka commented 1 year ago

@SYTHIER-ADS Was this issue provided to the IPF provider? If yes, what was the answer to this issue?

Woljtek commented 1 year ago

The issue PSC-67 has been created on ESA side. We are waiting for feedback to go further.

pcuq-ads commented 1 year ago

This is clearly a limitation for the system.

suberti-ads commented 9 months ago

PDGSANOM-14095 created

suberti-ads commented 9 months ago

For information this issue occurred on both satellite

suberti-ads commented 6 months ago

New occurrence 19 errors for SY_2_SYN products in PUG STC

[code 290] [exitCode 255] [msg Task /usr/local/components/PUG-3.48/bin/PUGCoreProcessor failed]

error

2024-04-12T01:20:31+00:00   2024-04-12T01:20:31.707479 s3-pug-stc-part1-execution-worker-v12-d7546576c-x6l6d PUG_SY_2_SYN 03.48 [0000000107]: [I] PUGCoreProcessor: Exiting with EXIT CODE: 255
2024-04-12T01:20:31+00:00       FATAL: All the product data unit generations exited in error!
2024-04-12T01:20:31+00:00       Size mismatching!2024-04-12T01:20:31.707379 s3-pug-stc-part1-execution-worker-v12-d7546576c-x6l6d PUG_SY_2_SYN 03.48 [0000000107]: [E] PUGCoreProcessor: [PUGCoreProcessor.C: execute:(359)] Unable to generate the required PDUs! --- acs::exCriticalException in PDUGenerator.C(270) from void acs::PDUGenerator::createPDUs() thread "" [139970633246912]
2024-04-12T01:20:31+00:00       acs::Netcdf4Copy::exNetcdf4CopyCriticalException in Netcdf4Copy.C(418) from static void acs::Netcdf4Copy::concatVars(const netCDF::NcVar&, const netCDF::NcVar&, netCDF::NcVar&, const size_t&) thread "unnamedThread" [139970183141120]
2024-04-12T01:20:31+00:00       caused by:
2024-04-12T01:20:31+00:00       Problem found during product data unit generation -> skipping to the next, if any ...
2024-04-12T01:20:31+00:00   acs::PDUGenerator::exPDUGeneratorException in (0) from  thread "" [139970633246912]

For information PDGSANMO-14095 created

COPRS / rs-issues

[BUG] S3-PUG-STC IPF crashes on some S3A_SY_2_SYN products #1033