usedatabrew / blink

OpenSource data platform to build event-driven systems. It's like Deebezium for golang :)
https://docs.databrew.tech/open-source/prerequisites
MIT License
22 stars 1 forks source link

feat(): working on nested fields #17

Closed le-vlad closed 8 months ago

le-vlad commented 8 months ago

Change Type

Define the type of the change your pull request will bring to the repo

Breaking Changes

Does your pull request introduce breaking changes?

Description

This PR brings the ability to work with nested fields for sources. Like it gives us an ability to consume data from various APIs and query structures like

{
   "data":{
      "array":[
         1,
         3,
         1
      ],
      "obj":{
         "name":"example"
      }
   }
}

Also, Mongodb snapshot streaming improvements. Added snapshot event type so we can use sink batch inserts for mongodb or postgresql to perform initial load faster

Testing

Describe how you have tested the changes or how we should test the changes:

Checklist

Please confirm the following:

Reviewers

Please add any appropriate reviewers for your PR.

Additional context

This PR also brings some small performance issues, like redundant JSON marshaling and unmarshaling for certain data Sources. I'm going to fix that later

Tests performed

During the testing, I checked the following combinations:

Postgres -> Postgres (snapshot, update, delete, insert)👌🏻  
Postgres -> MongoDB (snapshot, update, delete) 👌🏻 
WebSockets -> MongoDB  (upsert by default)👌🏻 
le-vlad commented 8 months ago

Postgres -> Postgres Replication is broken due to invalid SQL statements generated

Fixed statements generation. Snapshot and updates work as expected