flipp-oss / deimos

Framework to work with Kafka, Avro and ActiveRecord
Other
59 stars 22 forks source link

CCOL-2440: Add consumer configuration to save associated keys prior to saving primary record #217

Closed lionelpereira closed 1 month ago

lionelpereira commented 1 month ago

Pull Request Template

Description

Normal bulk consumption logic:

Widget:
has_one : detail
  1. Record class records (Widget) are saved
  2. Record class primary keys are filled from DB
  3. Associated records (Details) are updated with foreign key from record class

Backfill consumption logic (this change)

FidgetWidget
belongs_to: widget
belong_to: fidget

Notes:

  1. Associated records (Widget & Fidget) are created and linked to record class (FidgetWidget)
  2. Associated records are saved
  3. Primary keys (widget_id, fidget_id) of associated records are filled from DB
  4. Foreign keys of record class are filled with newly saved primary keys of associated records

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

Checklist:

dorner commented 1 month ago

@lionelpereira can you provide some more information in this PR as to what the use case is here and why it isn't covered by existing functionality?

lionelpereira commented 1 month ago

@lionelpereira can you provide some more information in this PR as to what the use case is here and why it isn't covered by existing functionality?

@dorner Updated the PR description