search

NASA-IMPACT / COSMOS

COSMOS is a web application designed to manage collections indexed in NASA's Science Discovery Engine (SDE), facilitating precise content selection and allowing metadata modification before indexing.
https://sde-indexing-helper.nasa-impact.net/
3 stars 1 forks source link

Pagination on the Sinequa sql.engine Api #1097

Open CarsonDavis opened 6 days ago

CarsonDavis commented 6 days ago

Description

Implementation Considerations

Deliverable

Dependencies

depends on #

### Acceptance Criteria
- [ ] Criteria 1
saifrk commented 3 days ago

-Docker Configuration Changes: Updated CPU and memory limits and reservations to the celeryworker service in Docker Compose -SQL Query Functionality Enhancements: -Pagination in SQL Queries: Modified the sql_query method to support pagination, which helps in handling large datasets by fetching data in chunks of 5000 records at a time. -Dynamic SQL Construction: Introduced SKIP and COUNT clauses dynamically into the SQL queries based on pagination -Data Processing: Added a method process_and_update_data to handle batch data processing and updating operations -Error Handling: Improved error handling in data processing to manage exceptions more effectively, specifically focusing on key errors and general exceptions. -Updated the fetch_and_replace_full_text function to perform Integrated Fetching and Processing, and also the soft_time_limit of the celery task has been increased.