FR: [Ingestion] - Synchronize parts information with OpenSearch index (6 of 6)

[urtzi-sotes]

Context/Description

The OpenSearch index must be synchronized with the parts' information whenever a part is created, updated or deleted. Thus, we need to implement a mechanism to perform the synchronization. Every time a file is published or unpublished, a repository_dispatch event is sent to the GitHub repo. We can create a GitHub action to trigger the ingestion pipeline whenever one of the parts' files is published:

• /cross-reference-data/cr-data.json
• /product-data/road-choice-make-model-part-filter-options.json
• /product-images/road-choice-website-images.json

We must ensure that all events are processed sequentially, e.g., by including a queue service in the beginning of the ingestion pipeline.

Acceptance Criteria

• The mechanism to perform synchronization is defined and documented in the wiki.
• The mechanism is implemented.
• The ingestion pipeline is triggered every time one of the related files is published as preview.
• end to end testing:
  • Every time a part file is published, the pipeline is triggered via de GitHub action.
  • As a result, the OpenSearch index is updated and:
  • new products are searchable.
  • deleted products are no longer searchable.
  • updated products contain the updated values

Out of scope

• Queue service to process all events sequentially

[ZoeBeykirch]

All ingestion tickets need to be completed before this one can be picked up.

[ZoeBeykirch]

Once this is completed, then can deploy the ingestion pipeline to production.