COSMOS is a web application designed to manage collections indexed in NASA's Science Discovery Engine (SDE), facilitating precise content selection and allowing metadata modification before indexing.
-Docker Configuration Changes: Updated CPU and memory limits and reservations to the celeryworker service in Docker Compose
-SQL Query Functionality Enhancements:
-Pagination in SQL Queries: Modified the sql_query method to support pagination, which helps in handling large datasets by fetching
data in chunks of 5000 records at a time.
-Dynamic SQL Construction: Introduced SKIP and COUNT clauses dynamically into the SQL queries based on pagination
-Data Processing: Added a method process_and_update_data to handle batch data processing and updating operations
-Error Handling: Improved error handling in data processing to manage exceptions more effectively, specifically focusing on key
errors and general exceptions.
-Updated the fetch_and_replace_full_text function to perform Integrated Fetching and Processing, and also the soft_time_limit of the celery task has been increased.
-Docker Configuration Changes: Updated CPU and memory limits and reservations to the celeryworker service in Docker Compose -SQL Query Functionality Enhancements: -Pagination in SQL Queries: Modified the sql_query method to support pagination, which helps in handling large datasets by fetching data in chunks of 5000 records at a time. -Dynamic SQL Construction: Introduced SKIP and COUNT clauses dynamically into the SQL queries based on pagination -Data Processing: Added a method process_and_update_data to handle batch data processing and updating operations -Error Handling: Improved error handling in data processing to manage exceptions more effectively, specifically focusing on key errors and general exceptions. -Updated the fetch_and_replace_full_text function to perform Integrated Fetching and Processing, and also the soft_time_limit of the celery task has been increased.