search

NASA-PDS / nucleus

Nucleus is a software platform used to create workflows for the Planetary Data (PDS).
https://nasa-pds.github.io/nucleus
Apache License 2.0
0 stars 0 forks source link

Nucleus Airflow Validate task fails with error while reading data file: For input string: "Infinity" #87

Closed ramesh-maddegoda closed 3 weeks ago

ramesh-maddegoda commented 9 months ago

Checked for duplicates

Yes - I've already checked

🐛 Describe the bug

When Nucleus executed the Airflow Validate task for some of the product labels in messenger MSGRMDS_8001 directory, that failed with the following error. 

FAIL: file:/mnt/data/pds-nucleus-staging/messenger-data/MSGRMDS_8001/RTM/MDIS_RTM_W11/2013_230/MDIS_RTM_W11_006680_4656687_1.xml
[2024-02-22, 19:24:38 UTC] {{task_log_fetcher.py:65}} INFO - [2024-02-22, 19:24:34 UTC]     Begin Content Validation: file:/mnt/data/pds-nucleus-staging/messenger-data/MSGRMDS_8001/RTM/MDIS_RTM_W11/2013_230/MDIS_RTM_W11_006680_4656687_1.IMG
[2024-02-22, 19:24:38 UTC] {{task_log_fetcher.py:65}} INFO - [2024-02-22, 19:24:34 UTC]       FATAL_ERROR  [error.array.bad_file_read]   array 1: Error occurred while reading data file: For input string: "Infinity"
[2024-02-22, 19:24:38 UTC] {{task_log_fetcher.py:65}} INFO - [2024-02-22, 19:24:34 UTC]     End Content Validation: file:/mnt/data/pds-nucleus-staging/messenger-data/MSGRMDS_8001/RTM/MDIS_RTM_W11/2013_230/MDIS_RTM_W11_006680_4656687_1.IMG
[2024-02-22, 19:24:38 UTC] {{task_log_fetcher.py:65}} INFO - [2024-02-22, 19:24:34 UTC]         1 product validation(s) completed
[2024-02-22, 19:24:38 UTC] {{task_log_fetcher.py:65}} INFO - [2024-02-22, 19:24:35 UTC]   FAIL: file:/mnt/data/pds-nucleus-staging/messenger-data/MSGRMDS_8001/RTM/MDIS_RTM_W11/2013_234/MDIS_RTM_W11_006603_4677154_1.xml
[2024-02-22, 19:24:38 UTC] {{task_log_fetcher.py:65}} INFO - [2024-02-22, 19:24:35 UTC]     Begin Content Validation: file:/mnt/data/pds-nucleus-staging/messenger-data/MSGRMDS_8001/RTM/MDIS_RTM_W11/2013_234/MDIS_RTM_W11_006603_4677154_1.IMG
[2024-02-22, 19:24:38 UTC] {{task_log_fetcher.py:65}} INFO - [2024-02-22, 19:24:35 UTC]       FATAL_ERROR  [error.array.bad_file_read]   array 1: Error occurred while reading data file: For input string: "Infinity"
[2024-02-22, 19:24:38 UTC] {{task_log_fetcher.py:65}} INFO - [2024-02-22, 19:24:35 UTC]     End Content Validation: file:/mnt/data/pds-nucleus-staging/messenger-data/MSGRMDS_8001/RTM/MDIS_RTM_W11/2013_234/MDIS_RTM_W11_006603_4677154_1.IMG
[2024-02-22, 19:24:38 UTC] {{task_log_fetcher.py:65}} INFO - [2024-02-22, 19:24:35 UTC]         2 product validation(s) completed
[2024-02-22, 19:24:38 UTC] {{task_log_fetcher.py:65}} INFO - [2024-02-22, 19:24:36 UTC]   FAIL: file:/mnt/data/pds-nucleus-staging/messenger-data/MSGRMDS_8001/RTM/MDIS_RTM_W11/2013_234/MDIS_RTM_W11_006614_4677166_1.xml
[2024-02-22, 19:24:38 UTC] {{task_log_fetcher.py:65}} INFO - [2024-02-22, 19:24:36 UTC]     Begin Content Validation: file:/mnt/data/pds-nucleus-staging/messenger-data/MSGRMDS_8001/RTM/MDIS_RTM_W11/2013_234/MDIS_RTM_W11_006614_4677166_1.IMG
[2024-02-22, 19:24:38 UTC] {{task_log_fetcher.py:65}} INFO - [2024-02-22, 19:24:36 UTC]       FATAL_ERROR  [error.array.bad_file_read]   array 1: Error occurred while reading data file: For input string: "Infinity"
[2024-02-22, 19:24:38 UTC] {{task_log_fetcher.py:65}} INFO - [2024-02-22, 19:24:36 UTC]     End Content Validation: file:/mnt/data/pds-nucleus-staging/messenger-data/MSGRMDS_8001/RTM/MDIS_RTM_W11/2013_234/MDIS_RTM_W11_006614_4677166_1.IMG
[2024-02-22, 19:24:38 UTC] {{task_log_fetcher.py:65}} INFO - [2024-02-22, 19:24:36 UTC]         3 product validation(s) completed
[2024-02-22, 19:24:38 UTC] {{task_log_fetcher.py:65}} INFO - [2024-02-22, 19:24:37 UTC]   FAIL: file:/mnt/data/pds-nucleus-staging/messenger-data/MSGRMDS_8001/RTM/MDIS_RTM_W11/2013_235/MDIS_RTM_W11_006602_4687392_1.xml
[2024-02-22, 19:24:38 UTC] {{task_log_fetcher.py:65}} INFO - [2024-02-22, 19:24:37 UTC]     Begin Content Validation: file:/mnt/data/pds-nucleus-staging/messenger-data/MSGRMDS_8001/RTM/MDIS_RTM_W11/2013_235/MDIS_RTM_W11_006602_4687392_1.IMG
[2024-02-22, 19:24:38 UTC] {{task_log_fetcher.py:65}} INFO - [2024-02-22, 19:24:37 UTC]       FATAL_ERROR  [error.array.bad_file_read]   array 1: Error occurred while reading data file: For input string: "Infinity"
[2024-02-22, 19:24:38 UTC] {{task_log_fetcher.py:65}} INFO - [2024-02-22, 19:24:37 UTC]     End Content Validation: file:/mnt/data/pds-nucleus-staging/messenger-data/MSGRMDS_8001/RTM/MDIS_RTM_W11/2013_235/MDIS_RTM_W11_006602_4687392_1.IMG
[2024-02-22, 19:24:38 UTC] {{task_log_fetcher.py:65}} INFO - [2024-02-22, 19:24:37 UTC]         4 product validation(s) completed

🕵️ Expected behavior

I expected the Nucleus Airflow Validate task to be successful without any error while reading data file: For input string: "Infinity"

📜 To Reproduce

Execute Nucleus validate task with:

Targets [file:/mnt/data/pds-nucleus-staging/messenger-data/MSGRMDS_8001/RTM/MDIS_RTM_W11/2013_230/MDIS_RTM_W11_006680_4656687_1.xml, file:/mnt/data/pds-nucleus-staging/messenger-data/MSGRMDS_8001/RTM/MDIS_RTM_W11/2013_234/MDIS_RTM_W11_006603_4677154_1.xml, file:/mnt/data/pds-nucleus-staging/messenger-data/MSGRMDS_8001/RTM/MDIS_RTM_W11/2013_234/MDIS_RTM_W11_006614_4677166_1.xml, file:/mnt/data/pds-nucleus-staging/messenger-data/MSGRMDS_8001/RTM/MDIS_RTM_W11/2013_235/MDIS_RTM_W11_006602_4687392_1.xml, file:/mnt/data/pds-nucleus-staging/messenger-data/MSGRMDS_8001/RTM/MDIS_RTM_W11/2013_236/MDIS_RTM_W11_006330_4691499_1.xml, file:/mnt/data/pds-nucleus-staging/messenger-data/MSGRMDS_8001/RTM/MDIS_RTM_W11/2013_237/MDIS_RTM_W11_006316_4695584_1.xml, file:/mnt/data/pds-nucleus-staging/messenger-data/MSGRMDS_8001/RTM/MDIS_RTM_W11/2013_237/MDIS_RTM_W11_006605_4699681_1.xml, file:/mnt/data/pds-nucleus-staging/messenger-data/MSGRMDS_8001/RTM/MDIS_RTM_W11/2013_238/MDIS_RTM_W11_006319_4701728_1.xml, file:/mnt/data/pds-nucleus-staging/messenger-data/MSGRMDS_8001/RTM/MDIS_RTM_W11/2013_238/MDIS_RTM_W11_006336_4701739_1.xml, file:/mnt/data/pds-nucleus-staging/messenger-data/MSGRMDS_8001/RTM/MDIS_RTM_W11/2013_244/MDIS_RTM_W11_006333_4740640_1.xml]

🖥 Environment Info

📚 Version of Software Used

No response

🩺 Test Data / Additional context

No response

🦄 Related requirements

🦄 #xyz

⚙️ Engineering Details

No response

tloubrieu-jpl commented 8 months ago

We decidedo skip the failed validation but there is another time out error on another directory.

jordanpadams commented 3 weeks ago

Closing this as wontfix. The issue with the data is expected. We need to continue processing data.