search

COPRS / rs-issues

This repository contains all the issues of the COPRS project (Scrum tickets, ivv bugs, epics ...)
2 stars 2 forks source link

[BUG] [Distribution v1.13.1-rc1+WA982] Fails to extract OL1 / OL2 products generated by PUG #1028

Open Woljtek opened 1 year ago

Woljtek commented 1 year ago

Environment:

Current Behavior: The distribution-worker fails to extract prip-metadata for:

Expected Behavior: Distribution shall be able to publish all products

Steps To Reproduce: Republish in compression-event the message related to once of the 4 OL1_EFR products. 001_1440_LN3_O_NR_004

Test execution artefacts (i.e. logs, screenshots…) The current error is the following: 

Error on publishing file S3A_OL_1_EFR____20230527T214047_20230527T222513_20230707T012733_2666_099_186______LN3_D_NT_002.SEN3 in PRIP: java.lang.RuntimeException: Error: Number of retries has exceeded while performing saving prip metadata of S3A_OL_1_EFR____20230527T214047_20230527T222513_20230707T012733_2666_099_186______LN3_D_NT_002.SEN3.zip after 4 attempts: ElasticsearchStatusException[Elasticsearch exception [type=mapper_parsing_exception, reason=failed to parse]]; nested: ElasticsearchException[Elasticsearch exception [type=topology_exception, reason=topology_exception: found non-noded intersection between LINESTRING ( -135.579 72.7312, -132.991 73.5004 ) and LINESTRING ( -1.7874499999999784 73.0118, -180.0 73.1260906895317 ) [ (-134.3488795168782, 73.09681386229416, NaN) ]]];
    at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:176)
    at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2011)
    at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1988)
    at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1745)
    at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1702)

Bug Generic Definition of Ready (DoR)

Bug Generic Definition of Done (DoD)

Woljtek commented 1 year ago

Note: The error message is identical to error message in METADATA => [BUG] [Metadata v1.13.2-rc1] Fails to extract 1 specific SL_1 and 1 SL_2 product generated by PUG #1002

w-jka commented 1 year ago

@Woljtek The WA for #1002 was only implemented in the metadata-extraction so far, as the distribution-worker will only copy the footprint from the extracted product of the metadata-extraction. If the save of the footprint fails on the metadata-extraction end, it won't contain the footprint, and will therefore not run into any issue in the distribution worker - so the current thought process behind the implementation.

I couldn't find the mentioned products in the s3_pug index but only in the s3_l1_ntc index, so I assume that those products might be the ones you mentioned in this issue. Those products do contain a footprint in the index, so the workaround of #1002 was not applied. I could not find any specific discrepancy between the configuration of the mappings for the coordinates so I am a bit lost, as to why ES allows the footprint in the s3_l1_ntc but throws an error on the prip one.

Could you provide the logs of the distribution-worker for further analysis? I could also add the workaround of #1002 again to the distribution-worker but the logs might help to understand what is happening here.

LAQU156 commented 1 year ago

IVV_CCB_2023_w28 : Moved into "Accepted Werum" for correction, Action on @Woljtek side : Please give logs about this issue. Priority blocking, to be fixed phase 1

LAQU156 commented 1 year ago

Werum_CCB_2023_w28 : Moved into "Product Backlog", waiting for logs from @Woljtek

Woljtek commented 1 year ago

@w-jka I extracted the log around the error message for S3A_OL_1_EFR__20230527T214047_20230527T222513_20230707T012733_2666_099_186____LN3_D_NT_002.SEN3.zip: Log in LOKI-logs-2023-07-12 16 32 54.txt

w-jka commented 1 year ago

@Woljtek The big issue with this bug is, that the metadata-extraction is working fine, while the distribution-worker is having problems with the same footprints. I could reproduce this error in a small testing environment using the logs you provided yesterday and could identify the reason as to why this behaviour happens:

Test scenario 

docker run --rm -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.15.2

This commands starts a single-node elasticsearch instance locally, which is afterwards available on localhost:9200.

curl -X PUT 'localhost:9200/test1' -H 'Content-Type: application/json' -d '{"mappings": { "properties": { "footprint": { "type": "geo_shape" }}}}'

curl -X PUT 'localhost:9200/test2' -H 'Content-Type: application/json' -d '{"mappings": { "properties": { "footprint": { "type": "geo_shape", "strategy" : "recursive", "tree" : "geohash" }}}}'

These two commands create two indices, test1 containing the configuration of the metadata-extraction indices, and test2 containing the configuration of the trip index for the footprint field.

curl -X POST 'localhost:9200/test1/_doc' -H 'Content-Type: application/json' -d '{"footprint":{"orientation":"counterclockwise","coordinates":[[[6.71086,73.2704],[6.59247,74.3056],[6.48148,75.3406],[6.37688,76.3754],[6.27836,77.4103],[6.19037,78.4451],[6.11607,79.4798],[6.05873,80.5141],[6.02545,81.5484],[6.02668,82.5828],[6.07829,83.6169],[6.20854,84.6506],[6.47425,85.684],[7.00462,86.7175],[8.15619,87.7498],[11.5158,88.7788],[46.5131,89.7599],[172.764,89.1264],[178.455,88.1006],[-179.957,87.069],[-179.276,86.0359],[-178.942,85.0024],[-178.776,83.9687],[-178.702,82.9348],[-178.687,81.9007],[-178.71,80.8664],[-178.76,79.8318],[-178.828,78.7973],[-178.911,77.7628],[-179.006,76.7279],[-179.109,75.6929],[-179.219,74.658],[-179.335,73.6228],[-179.456,72.5875],[-179.581,71.552],[-179.71,70.5163],[-179.843,69.4806],[-179.977,68.4444],[179.888,67.408],[179.75,66.3719],[179.61,65.3354],[179.467,64.2987],[179.323,63.2619],[179.179,62.2246],[179.032,61.187],[178.885,60.1492],[178.736,59.1115],[178.586,58.0736],[178.435,57.0352],[178.283,55.9967],[178.13,54.9577],[177.976,53.9186],[177.821,52.8797],[177.666,51.8403],[177.509,50.8003],[177.351,49.7604],[177.192,48.7204],[177.032,47.6801],[176.871,46.639],[176.71,45.5978],[176.548,44.5565],[176.384,43.5153],[176.22,42.4737],[176.054,41.4316],[175.888,40.3894],[175.721,39.347],[175.552,38.3047],[175.383,37.2618],[175.212,36.2185],[175.041,35.1751],[174.868,34.1316],[174.694,33.088],[174.519,32.0443],[174.343,31.0003],[174.165,29.956],[173.987,28.9116],[173.807,27.8672],[173.625,26.8224],[173.443,25.7772],[173.259,24.7322],[173.074,23.6873],[172.887,22.6423],[172.698,21.597],[172.508,20.5513],[172.317,19.5057],[172.124,18.4604],[171.929,17.4149],[171.732,16.3693],[171.534,15.3236],[171.334,14.2779],[171.132,13.2325],[170.927,12.1869],[170.721,11.1417],[170.513,10.0963],[170.303,9.05044],[170.091,8.00506],[169.876,6.96018],[169.658,5.91544],[169.438,4.87067],[169.216,3.82578],[168.991,2.78122],[168.764,1.73708],[168.534,0.693399],[168.3,-0.349973],[168.064,-1.39323],[167.824,-2.43635],[167.582,-3.47922],[167.336,-4.52144],[167.086,-5.56305],[166.833,-6.60426],[166.576,-7.64515],[166.315,-8.68556],[166.05,-9.72536],[165.782,-10.7649],[165.509,-11.8038],[165.231,-12.8416],[164.948,-13.8784],[164.66,-14.9148],[164.367,-15.9509],[164.07,-16.9862],[163.766,-18.0205],[163.457,-19.0537],[163.141,-20.0858],[162.819,-21.1171],[162.491,-22.1476],[162.156,-23.1769],[161.813,-24.2047],[161.463,-25.2311],[161.105,-26.2567],[160.739,-27.2813],[160.364,-28.3042],[159.98,-29.325],[159.586,-30.3445],[159.182,-31.3623],[158.768,-32.3788],[158.342,-33.3932],[157.905,-34.4055],[157.455,-35.4158],[156.992,-36.4238],[156.516,-37.4299],[156.025,-38.4338],[155.519,-39.4344],[154.996,-40.4321],[154.456,-41.4272],[153.898,-42.4199],[153.321,-43.4092],[152.723,-44.3943],[152.103,-45.3755],[151.459,-46.3533],[150.792,-47.3273],[150.615,-47.5785],[153.139,-48.357],[155.738,-49.0783],[158.41,-49.7393],[161.152,-50.337],[163.959,-50.8684],[166.827,-51.3308],[167.795,-51.4691],[168.157,-50.4307],[168.509,-49.3922],[168.851,-48.3536],[169.184,-47.3138],[169.508,-46.2725],[169.824,-45.23],[170.133,-44.1876],[170.436,-43.1455],[170.733,-42.1024],[171.023,-41.0576],[171.307,-40.0122],[171.588,-38.9677],[171.863,-37.9219],[172.135,-36.8758],[172.402,-35.8291],[172.665,-34.7815],[172.925,-33.7339],[173.182,-32.6862],[173.436,-31.6382],[173.687,-30.5892],[173.935,-29.5391],[174.181,-28.4892],[174.425,-27.4398],[174.667,-26.3896],[174.907,-25.3388],[175.146,-24.2875],[175.382,-23.2363],[175.618,-22.185],[175.852,-21.1331],[176.085,-20.0804],[176.317,-19.0273],[176.548,-17.9742],[176.779,-16.9216],[177.009,-15.8695],[177.239,-14.8167],[177.468,-13.7626],[177.696,-12.708],[177.924,-11.654],[178.153,-10.6001],[178.381,-9.54598],[178.61,-8.49199],[178.839,-7.43803],[179.068,-6.38359],[179.297,-5.32853],[179.527,-4.27356],[179.758,-3.21942],[179.99,-2.16511],[-179.778,-1.11069],[-179.545,-0.056098],[-179.311,0.998682],[-179.076,2.05349],[-178.839,3.10765],[-178.601,4.16141],[-178.361,5.21529],[-178.12,6.26976],[-177.878,7.32464],[-177.633,8.37858],[-177.386,9.43121],[-177.137,10.4843],[-176.886,11.5373],[-176.633,12.5903],[-176.377,13.6431],[-176.119,14.6952],[-175.857,15.7467],[-175.592,16.7979],[-175.324,17.8493],[-175.054,18.9005],[-174.779,19.9507],[-174.5,20.9999],[-174.218,22.0491],[-173.931,23.0981],[-173.64,24.147],[-173.345,25.1949],[-173.044,26.2417],[-172.738,27.2879],[-172.426,28.3337],[-172.109,29.3786],[-171.785,30.4227],[-171.455,31.4661],[-171.118,32.5087],[-170.773,33.5504],[-170.42,34.5912],[-170.06,35.631],[-169.691,36.6694],[-169.312,37.7069],[-168.923,38.7434],[-168.524,39.7787],[-168.114,40.8123],[-167.692,41.8445],[-167.257,42.8755],[-166.808,43.9051],[-166.345,44.9331],[-165.866,45.9585],[-165.371,46.9815],[-164.858,48.0032],[-164.325,49.0234],[-163.773,50.0411],[-163.198,51.0556],[-162.599,52.0676],[-161.974,53.077],[-161.321,54.0827],[-160.638,55.0855],[-159.921,56.0842],[-159.169,57.0788],[-158.378,58.0696],[-157.545,59.0555],[-156.665,60.0365],[-155.734,61.0109],[-154.747,61.9788],[-153.698,62.9402],[-152.581,63.8938],[-151.39,64.8393],[-150.116,65.7754],[-148.749,66.6995],[-147.28,67.6121],[-145.698,68.511],[-143.989,69.3948],[-142.14,70.2614],[-140.134,71.1082],[-137.954,71.9326],[-135.579,72.7312],[-132.991,73.5004],[-130.167,74.2361],[-127.086,74.9332],[-123.727,75.586],[-120.074,76.1883],[-116.115,76.7334],[-111.848,77.2138],[-107.282,77.6224],[-102.442,77.9517],[-97.3725,78.195],[-92.1362,78.3468],[-86.8079,78.4034],[-81.4719,78.3636],[-76.2127,78.2285],[-71.1086,78.0015],[-66.2242,77.6871],[-61.6072,77.2921],[-57.2864,76.8239],[-53.273,76.2896],[-49.5656,75.6968],[-46.1556,75.0524],[-43.0265,74.363],[-40.1581,73.6339],[-37.5289,72.8702],[-35.1175,72.0766],[-32.9052,71.2569],[-30.8716,70.4141],[-28.9957,69.551],[-28.5341,69.3248],[-24.0301,70.3461],[-19.0838,71.2437],[-13.6996,71.9993],[-7.91098,72.5942],[-1.78745,73.0118],[4.56514,73.2386],[6.71086,73.2704]]],"type":"polygon"}}'

--> {"_index":"test1","_type":"_doc","_id":"5o19TokBIlO8xbWLCYGK","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}

When publishing the footprint on test1 we receive a successful ingestion. This behaviour is identical to the one on the ops env.

curl -X POST 'localhost:9200/test2/_doc' -H 'Content-Type: application/json' -d '{"footprint":{"orientation":"counterclockwise","coordinates":[[[6.71086,73.2704],[6.59247,74.3056],[6.48148,75.3406],[6.37688,76.3754],[6.27836,77.4103],[6.19037,78.4451],[6.11607,79.4798],[6.05873,80.5141],[6.02545,81.5484],[6.02668,82.5828],[6.07829,83.6169],[6.20854,84.6506],[6.47425,85.684],[7.00462,86.7175],[8.15619,87.7498],[11.5158,88.7788],[46.5131,89.7599],[172.764,89.1264],[178.455,88.1006],[-179.957,87.069],[-179.276,86.0359],[-178.942,85.0024],[-178.776,83.9687],[-178.702,82.9348],[-178.687,81.9007],[-178.71,80.8664],[-178.76,79.8318],[-178.828,78.7973],[-178.911,77.7628],[-179.006,76.7279],[-179.109,75.6929],[-179.219,74.658],[-179.335,73.6228],[-179.456,72.5875],[-179.581,71.552],[-179.71,70.5163],[-179.843,69.4806],[-179.977,68.4444],[179.888,67.408],[179.75,66.3719],[179.61,65.3354],[179.467,64.2987],[179.323,63.2619],[179.179,62.2246],[179.032,61.187],[178.885,60.1492],[178.736,59.1115],[178.586,58.0736],[178.435,57.0352],[178.283,55.9967],[178.13,54.9577],[177.976,53.9186],[177.821,52.8797],[177.666,51.8403],[177.509,50.8003],[177.351,49.7604],[177.192,48.7204],[177.032,47.6801],[176.871,46.639],[176.71,45.5978],[176.548,44.5565],[176.384,43.5153],[176.22,42.4737],[176.054,41.4316],[175.888,40.3894],[175.721,39.347],[175.552,38.3047],[175.383,37.2618],[175.212,36.2185],[175.041,35.1751],[174.868,34.1316],[174.694,33.088],[174.519,32.0443],[174.343,31.0003],[174.165,29.956],[173.987,28.9116],[173.807,27.8672],[173.625,26.8224],[173.443,25.7772],[173.259,24.7322],[173.074,23.6873],[172.887,22.6423],[172.698,21.597],[172.508,20.5513],[172.317,19.5057],[172.124,18.4604],[171.929,17.4149],[171.732,16.3693],[171.534,15.3236],[171.334,14.2779],[171.132,13.2325],[170.927,12.1869],[170.721,11.1417],[170.513,10.0963],[170.303,9.05044],[170.091,8.00506],[169.876,6.96018],[169.658,5.91544],[169.438,4.87067],[169.216,3.82578],[168.991,2.78122],[168.764,1.73708],[168.534,0.693399],[168.3,-0.349973],[168.064,-1.39323],[167.824,-2.43635],[167.582,-3.47922],[167.336,-4.52144],[167.086,-5.56305],[166.833,-6.60426],[166.576,-7.64515],[166.315,-8.68556],[166.05,-9.72536],[165.782,-10.7649],[165.509,-11.8038],[165.231,-12.8416],[164.948,-13.8784],[164.66,-14.9148],[164.367,-15.9509],[164.07,-16.9862],[163.766,-18.0205],[163.457,-19.0537],[163.141,-20.0858],[162.819,-21.1171],[162.491,-22.1476],[162.156,-23.1769],[161.813,-24.2047],[161.463,-25.2311],[161.105,-26.2567],[160.739,-27.2813],[160.364,-28.3042],[159.98,-29.325],[159.586,-30.3445],[159.182,-31.3623],[158.768,-32.3788],[158.342,-33.3932],[157.905,-34.4055],[157.455,-35.4158],[156.992,-36.4238],[156.516,-37.4299],[156.025,-38.4338],[155.519,-39.4344],[154.996,-40.4321],[154.456,-41.4272],[153.898,-42.4199],[153.321,-43.4092],[152.723,-44.3943],[152.103,-45.3755],[151.459,-46.3533],[150.792,-47.3273],[150.615,-47.5785],[153.139,-48.357],[155.738,-49.0783],[158.41,-49.7393],[161.152,-50.337],[163.959,-50.8684],[166.827,-51.3308],[167.795,-51.4691],[168.157,-50.4307],[168.509,-49.3922],[168.851,-48.3536],[169.184,-47.3138],[169.508,-46.2725],[169.824,-45.23],[170.133,-44.1876],[170.436,-43.1455],[170.733,-42.1024],[171.023,-41.0576],[171.307,-40.0122],[171.588,-38.9677],[171.863,-37.9219],[172.135,-36.8758],[172.402,-35.8291],[172.665,-34.7815],[172.925,-33.7339],[173.182,-32.6862],[173.436,-31.6382],[173.687,-30.5892],[173.935,-29.5391],[174.181,-28.4892],[174.425,-27.4398],[174.667,-26.3896],[174.907,-25.3388],[175.146,-24.2875],[175.382,-23.2363],[175.618,-22.185],[175.852,-21.1331],[176.085,-20.0804],[176.317,-19.0273],[176.548,-17.9742],[176.779,-16.9216],[177.009,-15.8695],[177.239,-14.8167],[177.468,-13.7626],[177.696,-12.708],[177.924,-11.654],[178.153,-10.6001],[178.381,-9.54598],[178.61,-8.49199],[178.839,-7.43803],[179.068,-6.38359],[179.297,-5.32853],[179.527,-4.27356],[179.758,-3.21942],[179.99,-2.16511],[-179.778,-1.11069],[-179.545,-0.056098],[-179.311,0.998682],[-179.076,2.05349],[-178.839,3.10765],[-178.601,4.16141],[-178.361,5.21529],[-178.12,6.26976],[-177.878,7.32464],[-177.633,8.37858],[-177.386,9.43121],[-177.137,10.4843],[-176.886,11.5373],[-176.633,12.5903],[-176.377,13.6431],[-176.119,14.6952],[-175.857,15.7467],[-175.592,16.7979],[-175.324,17.8493],[-175.054,18.9005],[-174.779,19.9507],[-174.5,20.9999],[-174.218,22.0491],[-173.931,23.0981],[-173.64,24.147],[-173.345,25.1949],[-173.044,26.2417],[-172.738,27.2879],[-172.426,28.3337],[-172.109,29.3786],[-171.785,30.4227],[-171.455,31.4661],[-171.118,32.5087],[-170.773,33.5504],[-170.42,34.5912],[-170.06,35.631],[-169.691,36.6694],[-169.312,37.7069],[-168.923,38.7434],[-168.524,39.7787],[-168.114,40.8123],[-167.692,41.8445],[-167.257,42.8755],[-166.808,43.9051],[-166.345,44.9331],[-165.866,45.9585],[-165.371,46.9815],[-164.858,48.0032],[-164.325,49.0234],[-163.773,50.0411],[-163.198,51.0556],[-162.599,52.0676],[-161.974,53.077],[-161.321,54.0827],[-160.638,55.0855],[-159.921,56.0842],[-159.169,57.0788],[-158.378,58.0696],[-157.545,59.0555],[-156.665,60.0365],[-155.734,61.0109],[-154.747,61.9788],[-153.698,62.9402],[-152.581,63.8938],[-151.39,64.8393],[-150.116,65.7754],[-148.749,66.6995],[-147.28,67.6121],[-145.698,68.511],[-143.989,69.3948],[-142.14,70.2614],[-140.134,71.1082],[-137.954,71.9326],[-135.579,72.7312],[-132.991,73.5004],[-130.167,74.2361],[-127.086,74.9332],[-123.727,75.586],[-120.074,76.1883],[-116.115,76.7334],[-111.848,77.2138],[-107.282,77.6224],[-102.442,77.9517],[-97.3725,78.195],[-92.1362,78.3468],[-86.8079,78.4034],[-81.4719,78.3636],[-76.2127,78.2285],[-71.1086,78.0015],[-66.2242,77.6871],[-61.6072,77.2921],[-57.2864,76.8239],[-53.273,76.2896],[-49.5656,75.6968],[-46.1556,75.0524],[-43.0265,74.363],[-40.1581,73.6339],[-37.5289,72.8702],[-35.1175,72.0766],[-32.9052,71.2569],[-30.8716,70.4141],[-28.9957,69.551],[-28.5341,69.3248],[-24.0301,70.3461],[-19.0838,71.2437],[-13.6996,71.9993],[-7.91098,72.5942],[-1.78745,73.0118],[4.56514,73.2386],[6.71086,73.2704]]],"type":"polygon"}}'

--> {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse"}],"type":"mapper_parsing_exception","reason":"failed to parse","caused_by": {"type":"topology_exception","reason":"found non-noded intersection between LINESTRING ( -135.579 72.7312, -132.991 73.5004 ) and LINESTRING ( -1.7874499999999784 73.0118, -180.0 73.1260906895317 ) [ (-134.3488795168782, 73.09681386229416, NaN) ]"}},"status":400}

When publishing the same footprint to test2 we get an error 400, "non-noded intersection". This behaviour is also the same as on the ops env.

What is happening? The configuration of a strategy or tree property on a geo_shape field is changing the algorithm that is used by ES from a BKD-backed approach to an older algorithm (prefix tree). The prefix tree is deprecated with ES 7 (and removed in ES 8 and OpenSearch) and it is not advised to use it, as it is slower and uses more space. In our case additionally, the prefix tree has problems with footprints that are above the north and south pole. You can read more about this in the official documentation and blog post: https://www.elastic.co/guide/en/elasticsearch/reference/7.15/geo-shape.html#geo-shape-mapping-options https://www.elastic.co/de/blog/bkd-backed-geo-shapes-in-elasticsearch-precision-efficiency-speed

How can we change the algorithm? The migration can not be performed on the fly. A new index with the correct settings has to be created in order to use the correct algorithm.

w-jka commented 1 year ago

Documentation on how to perform a reindex of data from one index to another https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html

So in order to fix this issue one approach is:

  1. Undeploy distribution-chain
  2. Create a temporary prip-index (ex. prip_tmp)
  3. Reindex all data from prip to temporary prip-index
  4. Delete prip index and create new prip index with fixed configuration
  5. Reindex all data from temporary prip-index to new prip
  6. Deploy distribution-chain
  7. After everything is working as expected, delete temporary prip-index
w-jka commented 1 year ago

Candidate for refusal

pcuq-ads commented 1 year ago

Werum_CCB_2023_w30 : to be checked on ADS side :

pcuq-ads commented 1 year ago

System_CCB_2023-w30 : The RS Core - Elasticsearch Configuration is in line with the proposed fixed since the version 1.5 . The issue is still present in OPS, but a deployment from scratch will not have the issue. Priority reduced to minor with status CHECKED. See page : https://github.com/COPRS/production-common/blob/release/1.5.0/processing-common/doc/indices.md

suberti-ads commented 1 year ago

For information , this issue impacted also OL2 product for sample S3B_OL_2_LRR__20230603T212004_20230603T220430_20230707T125951_2666_080_143____LN3_D_NT_002

Error on publishing file S3B_OL_2_LRR____20230603T212004_20230603T220430_20230707T125951_2666_080_143______LN3_D_NT_002.SEN3 in PRIP: java.lang.RuntimeException: Error: Number of retries has exceeded while performing saving prip metadata of S3B_OL_2_LRR____20230603T212004_20230603T220430_20230707T125951_2666_080_143______LN3_D_NT_002.SEN3.zip after 4 attempts: ElasticsearchStatusException[Elasticsearch exception [type=mapper_parsing_exception, reason=failed to parse]]; nested: ElasticsearchException[Elasticsearch exception [type=topology_exception, reason=topology_exception: found non-noded intersection between LINESTRING ( -134.351 71.3427, -129.757 72.9269 ) and LINESTRING ( -1.8263900000000035 71.5222, -180.0 71.72491214102453 ) [ (-133.39638594926308, 71.67189015654711, NaN) ]]];
    at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:176)
    at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2011)
    at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1988)
    at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1745)
    at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1702)
    at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1672)
    at org.elasticsearch.client.RestHighLevelClient.index(RestHighLevelClient.java:1029)
    at esa.s1pdgs.cpoc.prip.metadata.PripElasticSearchMetadataRepo.save(PripElasticSearchMetadataRepo.java:101)
    at esa.s1pdgs.cpoc.prip.worker.service.PripPublishingService.lambda$createAndSave$0(PripPublishingService.java:198)
    at esa.s1pdgs.cpoc.common.utils.Retries.performWithRetries(Retries.java:23)
    at esa.s1pdgs.cpoc.prip.worker.service.PripPublishingService.createAndSave(PripPublishingService.java:197)
    at esa.s1pdgs.cpoc.prip.worker.service.PripPublishingService.accept(PripPublishingService.java:105)
    at esa.s1pdgs.cpoc.prip.worker.service.PripPublishingService.accept(PripPublishingService.java:56)
    at org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.invokeConsumer(SimpleFunctionRegistry.java:976)
    at org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.doApply(SimpleFunctionRegistry.java:705)
    at org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.apply(SimpleFunctionRegistry.java:551)
    at org.springframework.cloud.stream.function.PartitionAwareFunctionWrapper.apply(PartitionAwareFunctionWrapper.java:84)
    at org.springframework.cloud.stream.function.FunctionConfiguration$FunctionWrapper.apply(FunctionConfiguration.java:754)
    at org.springframework.cloud.stream.function.FunctionConfiguration$FunctionToDestinationBinder$1.handleMessageInternal(FunctionConfiguration.java:586)
    at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:56)
    at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:115)
    at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:133)
    at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:106)
    at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:72)
    at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:317)
    at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:272)
    at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:187)
    at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:166)
    at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:47)
    at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:109)
    at org.springframework.integration.endpoint.MessageProducerSupport.sendMessage(MessageProducerSupport.java:216)
    at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter.sendMessageIfAny(KafkaMessageDrivenChannelAdapter.java:397)
    at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter.access$300(KafkaMessageDrivenChannelAdapter.java:83)
    at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter$IntegrationRecordMessageListener.onMessage(KafkaMessageDrivenChannelAdapter.java:454)
    at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter$IntegrationRecordMessageListener.onMessage(KafkaMessageDrivenChannelAdapter.java:428)
    at org.springframework.kafka.listener.adapter.RetryingMessageListenerAdapter.lambda$onMessage$0(RetryingMessageListenerAdapter.java:125)
    at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:329)
    at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:255)
    at org.springframework.kafka.listener.adapter.RetryingMessageListenerAdapter.onMessage(RetryingMessageListenerAdapter.java:119)
    at org.springframework.kafka.listener.adapter.RetryingMessageListenerAdapter.onMessage(RetryingMessageListenerAdapter.java:42)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:2629)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeOnMessage(KafkaMessageListenerContainer.java:2609)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:2536)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeWithRecords(KafkaMessageListenerContainer.java:2427)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:2305)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:1979)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeIfHaveRecords(KafkaMessageListenerContainer.java:1364)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1355)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1247)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.lang.Thread.run(Thread.java:829)
    Suppressed: org.elasticsearch.client.ResponseException: method [PUT], host [http://elasticsearch-processing-es-http.database.svc.cluster.local:9200], URI [/prip/_doc/S3B_OL_2_LRR____20230603T212004_20230603T220430_20230707T125951_2666_080_143______LN3_D_NT_002.SEN3.zip?timeout=1m], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"topology_exception","reason":"topology_exception: found non-noded intersection between LINESTRING ( -134.351 71.3427, -129.757 72.9269 ) and LINESTRING ( -1.8263900000000035 71.5222, -180.0 71.72491214102453 ) [ (-133.39638594926308, 71.67189015654711, NaN) ]"}],"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"topology_exception","reason":"topology_exception: found non-noded intersection between LINESTRING ( -134.351 71.3427, -129.757 72.9269 ) and LINESTRING ( -1.8263900000000035 71.5222, -180.0 71.72491214102453 ) [ (-133.39638594926308, 71.67189015654711, NaN) ]"}},"status":400}
        at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:326)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270)
        at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2082)
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1732)
        ... 48 more
Caused by: ElasticsearchException[Elasticsearch exception [type=topology_exception, reason=topology_exception: found non-noded intersection between LINESTRING ( -134.351 71.3427, -129.757 72.9269 ) and LINESTRING ( -1.8263900000000035 71.5222, -180.0 71.72491214102453 ) [ (-133.39638594926308, 71.67189015654711, NaN) ]]]
    at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:485)
    at org.elasticsearch.ElasticsearchException.fromXContent(ElasticsearchException.java:396)
    at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:426)
    at org.elasticsearch.ElasticsearchException.failureFromXContent(ElasticsearchException.java:592)
    at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:168)
    ... 51 more

    at esa.s1pdgs.cpoc.common.utils.Retries.throwRuntimeException(Retries.java:53)
    at esa.s1pdgs.cpoc.common.utils.Retries.performWithRetries(Retries.java:28)
    at esa.s1pdgs.cpoc.prip.worker.service.PripPublishingService.createAndSave(PripPublishingService.java:197)
    at esa.s1pdgs.cpoc.prip.worker.service.PripPublishingService.accept(PripPublishingService.java:105)
    at esa.s1pdgs.cpoc.prip.worker.service.PripPublishingService.accept(PripPublishingService.java:56)
    at org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.invokeConsumer(SimpleFunctionRegistry.java:976)
    at org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.doApply(SimpleFunctionRegistry.java:705)
    at org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.apply(SimpleFunctionRegistry.java:551)
    at org.springframework.cloud.stream.function.PartitionAwareFunctionWrapper.apply(PartitionAwareFunctionWrapper.java:84)
    at org.springframework.cloud.stream.function.FunctionConfiguration$FunctionWrapper.apply(FunctionConfiguration.java:754)
    at org.springframework.cloud.stream.function.FunctionConfiguration$FunctionToDestinationBinder$1.handleMessageInternal(FunctionConfiguration.java:586)
    at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:56)
    at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:115)
    at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:133)
    at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:106)
    at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:72)
    at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:317)
    at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:272)
    at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:187)
    at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:166)
    at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:47)
    at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:109)
    at org.springframework.integration.endpoint.MessageProducerSupport.sendMessage(MessageProducerSupport.java:216)
    at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter.sendMessageIfAny(KafkaMessageDrivenChannelAdapter.java:397)
    at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter.access$300(KafkaMessageDrivenChannelAdapter.java:83)
    at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter$IntegrationRecordMessageListener.onMessage(KafkaMessageDrivenChannelAdapter.java:454)
    at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter$IntegrationRecordMessageListener.onMessage(KafkaMessageDrivenChannelAdapter.java:428)
    at org.springframework.kafka.listener.adapter.RetryingMessageListenerAdapter.lambda$onMessage$0(RetryingMessageListenerAdapter.java:125)
    at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:329)
    at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:255)
    at org.springframework.kafka.listener.adapter.RetryingMessageListenerAdapter.onMessage(RetryingMessageListenerAdapter.java:119)
    at org.springframework.kafka.listener.adapter.RetryingMessageListenerAdapter.onMessage(RetryingMessageListenerAdapter.java:42)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:2629)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeOnMessage(KafkaMessageListenerContainer.java:2609)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:2536)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeWithRecords(KafkaMessageListenerContainer.java:2427)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:2305)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:1979)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeIfHaveRecords(KafkaMessageListenerContainer.java:1364)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1355)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1247)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: ElasticsearchStatusException[Elasticsearch exception [type=mapper_parsing_exception, reason=failed to parse]]; nested: ElasticsearchException[Elasticsearch exception [type=topology_exception, reason=topology_exception: found non-noded intersection between LINESTRING ( -134.351 71.3427, -129.757 72.9269 ) and LINESTRING ( -1.8263900000000035 71.5222, -180.0 71.72491214102453 ) [ (-133.39638594926308, 71.67189015654711, NaN) ]]];
    at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:176)
    at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2011)
    at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1988)
    at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1745)
    at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1702)
    at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1672)
    at org.elasticsearch.client.RestHighLevelClient.index(RestHighLevelClient.java:1029)
    at esa.s1pdgs.cpoc.prip.metadata.PripElasticSearchMetadataRepo.save(PripElasticSearchMetadataRepo.java:101)
    at esa.s1pdgs.cpoc.prip.worker.service.PripPublishingService.lambda$createAndSave$0(PripPublishingService.java:198)
    at esa.s1pdgs.cpoc.common.utils.Retries.performWithRetries(Retries.java:23)
    ... 42 more
    Suppressed: org.elasticsearch.client.ResponseException: method [PUT], host [http://elasticsearch-processing-es-http.database.svc.cluster.local:9200], URI [/prip/_doc/S3B_OL_2_LRR____20230603T212004_20230603T220430_20230707T125951_2666_080_143______LN3_D_NT_002.SEN3.zip?timeout=1m], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"topology_exception","reason":"topology_exception: found non-noded intersection between LINESTRING ( -134.351 71.3427, -129.757 72.9269 ) and LINESTRING ( -1.8263900000000035 71.5222, -180.0 71.72491214102453 ) [ (-133.39638594926308, 71.67189015654711, NaN) ]"}],"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"topology_exception","reason":"topology_exception: found non-noded intersection between LINESTRING ( -134.351 71.3427, -129.757 72.9269 ) and LINESTRING ( -1.8263900000000035 71.5222, -180.0 71.72491214102453 ) [ (-133.39638594926308, 71.67189015654711, NaN) ]"}},"status":400}
        at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:326)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270)
        at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2082)
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1732)
        ... 48 more
Caused by: ElasticsearchException[Elasticsearch exception [type=topology_exception, reason=topology_exception: found non-noded intersection between LINESTRING ( -134.351 71.3427, -129.757 72.9269 ) and LINESTRING ( -1.8263900000000035 71.5222, -180.0 71.72491214102453 ) [ (-133.39638594926308, 71.67189015654711, NaN) ]]]
    at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:485)
    at org.elasticsearch.ElasticsearchException.fromXContent(ElasticsearchException.java:396)
    at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:426)
    at org.elasticsearch.ElasticsearchException.failureFromXContent(ElasticsearchException.java:592)
    at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:168)
    ... 51 more

I update issue title