Data structure matters that we need to pay attention

[PhVHoang]

• Lambda vs Kappa:

  • Lambda: more complex, you have a speed layer and a batch layer, the batch layer is more trustworthy, picks up low latency and correctness at the expense of complexity.
  • Kappa: Simpler, only streaming, data quality alerts, picks up latency and simplicity at the expense of correctness.

• Data correctness patter:

  • Signal table vs write-audit-push: https://blog.dataengineer.io/p/writing-data-to-production-is-a-contract
  • Test streaming pipelines for errors:
  • Fail the job on egregious errors to troubleshoot
  • Output the error rows to a separate Kafka topic
  • Ignore the error rows with filter conditions

• Tradeoffs of different database choices

References

https://blog.dataengineer.io/p/how-to-pass-the-data-architecture https://github.com/PhVHoang/TIL/issues/410