Unmatched assets between PG <> ES

[batrov]

Is your feature request related to a problem? Please describe. Currently, there is a discrepancy between the assets in PG and ES, and the reason for this is unknown. This discrepancy adversely affects search relevancy and user experience, as users are unable to find the assets they are looking for or are presented with assets that are no longer available.

Describe the solution you'd like Provide a Sync API to recreate ES indexes

• copy old index to temp index
• change the alias to temp index
• delete old index
• create new index
• sync the data
• change the alias to new index
• delete the temp index

High Level Flow:

We will only allow 1 sync job to run for 1 index to prevent the race conditions during the process

[bsushmith]

@batrov @haveiss

there is a discrepancy between the assets in PG and ES, and the reason for this is unknown

Can we try to identify the root cause of this?

I think this new API ,though quite helpful, addresses a different problem and not the root cause itself.

[haveiss]

@batrov @haveiss

there is a discrepancy between the assets in PG and ES, and the reason for this is unknown

Can we try to identify the root cause of this?

I think this new API ,though quite helpful, addresses a different problem and not the root cause itself.

a lot of factors that i can think of

• Tech debt from the past: This issue seems to stem from unresolved technical debt that has accumulated over time.
  • we had this issue https://github.com/goto/compass/pull/28
• Previous lack of robust asset synchronization: In the past, we did not have a reliable method to ensure that assets would be consistently synchronized between PG and ES. To address this, we have introduced an asynchronous worker that will eventually ensure consistency. (See: https://github.com/goto/compass/pull/45)
• There's possibility people with access updating data directly to PG