search

NASA-PDS / nucleus

Nucleus is a software platform used to create workflows for the Planetary Data (PDS).
https://nasa-pds.github.io/nucleus
Apache License 2.0
0 stars 0 forks source link

Nucleus lambda function pds-nucleus-datasync-completion sometimes fails to write copied data file names to Nucleus database #83

Closed ramesh-maddegoda closed 7 months ago

ramesh-maddegoda commented 7 months ago

Checked for duplicates

Yes - I've already checked

🐛 Describe the bug

When I copied a large amount data with AWS DataSync, the Nucleus lambda function pds-nucleus-datasync-completion got triggered as expected. Also, Nucleus was able to process a many products. However, I noticed that there are few products remain in the Nucleus database as "INCOMPLETE". When I checked the database and files, I had the following observations.

1) The AWS DataSyn has copied all the data files and products labels of these "INCOMPLETE" products to the EFS volume. 2) The Nucleus lambda function pds-nucleus-datasync-completion has successfully updated the Nucleus database tables: product and product_data_file_mapping 3) The Nucleus lambda function pds-nucleus-datasync-completion has failed to update the Nucleus database table: data_file 4) The failed data files that were not written to the database are large IMG files

🕵️ Expected behavior

I expected the Nucleus lambda function pds-nucleus-datasync-completion to update the Nucleus table: data_file, after copying the data file to EFS, regardless of the size or type of the data file.

📜 To Reproduce

Copy a large amount data with AWS DataSync and let the the Nucleus lambda function pds-nucleus-datasync-completion to get triggered.

Check if it has relevant records for all the files copied.

tloubrieu-jpl commented 7 months ago

Hi @ramesh-maddegoda ,

Do you foresee a PR associated with this ticket ? I am wondering what the review process will be here.

Thanks

ramesh-maddegoda commented 7 months ago

Hi @ramesh-maddegoda ,

Do you foresee a PR associated with this ticket ? I am wondering what the review process will be here.

Thanks

@tloubrieu-jpl , I changed the status to Review/QA, because I was doing QA on it with large data sets. Now QA is completed.

I also have a pull request now: https://github.com/NASA-PDS/nucleus/pull/85

ramesh-maddegoda commented 7 months ago

The pull request: https://github.com/NASA-PDS/nucleus/pull/85 approved and also tested successfully.