In addition to URLs, also bring in full-text from Sinequa servers

NASA-IMPACT / COSMOS

COSMOS is a web application designed to manage collections indexed in NASA's Science Discovery Engine (SDE), facilitating precise content selection and allowing metadata modification before indexing.

https://sde-indexing-helper.nasa-impact.net/

3 stars 1 forks source link

In addition to URLs, also bring in full-text from Sinequa servers #1016

Open code-geek opened 2 months ago

code-geek commented 2 months ago

Description

When we bring in URLs right now, we just get the URL and title. We also want to store the full-text in the database (but not show it in the table necessarily). This will allow us to track when fulltext for pages change, and also use this data for LLM purposes.

Implementation Considerations

Needs a new field in the CandidateURL model
Might need to preserve the last one so we can see deltas
Perhaps some sort of hashing could help us (MD5)

Deliverable

Notification when there are new URLs, new titles, new text, and a clear way to tell

Dependencies

No response

CarsonDavis commented 1 month ago

We will probably need to use the sql endpoint from sinequa. Documentation can be found at this link: https://doc.sinequa.com/en.sinequa-es.v11/Content/en.sinequa-es.devDoc.webservice.rest-search.html#engine-sql

CarsonDavis commented 1 month ago

Existing code using this endpoint can be found in the following files:

CarsonDavis commented 1 month ago

You may need a token from the server that includes SQL engine access. You can search for tokens in the admin console, and then you make a new token, and you give it permissions.