When the training job is executed do not create embeddings for already trained sections that are not updated.

[myrtp]

Description

When Re-Triggering the Jobs for a previous period, dont request embeddings for sections that already have, unless stated otherwise.

Hints

Add DocumentInstanceSectionCriteria field and respective Predicate function, that selects only updated sections with no embedding. Embedding could exist, but for other/previous version of section. Eg.

• DB at 2024-11-07 section:"section A v1" ---- Trigger job ----
• DB at 2024-11-08 section:"section A v1" -> embedding: [1,2,3]
• DB at 2024-11-09 section:"section B v2" -> embedding: [1,2,3] ---- Trigger job with flag ---- ---- sections has been updated ----
• DB at 2024-11-10 section:"section B v2" -> embedding: [7,8,9]

Select sections with subquery:

select *
from gendox_core.document_instance_sections dis
where dis.id in (select dis1.id
              from gendox_core.document_instance_sections dis1
                       inner join gendox_core.document_instance di on di.id = dis1.document_instance_id
                       inner join gendox_core.project_documents pd on di.id = pd.document_id
                       inner join gendox_core.project_agent pa on pd.project_id = pa.project_id
                       inner join gendox_core.embedding_group eg on dis1.id = eg.section_id
              where pa.semantic_search_model_id = eg.semantic_search_model_id and
                  dis1.updated_at > eg.updated_at);

Criterion needs to be added in DocumentInstanceSectionPredicates

example on how to add subquery in predicate: DocumentInstanceSectionPredicates#project this ⏫ , when added in the query, selects all sections, of projects that have autoTraining=true

Similarly we want a subquery that, selects all sections, with embedding older than the respective section (undate_at)

[myrtp]

this will be implemented by adding SHA256 codes on document and embedding group