The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
I believe we are already querying all the fields in the document - and then filtering them out based on the fields in the catalog. This change would just involve NOT filtering these fields out in packed mode
Initial Snapshot : Code Link. Then, we use this method to strip the fields not in the catalog. However, there is an override already defined to allow all fields.
Incremental Load (from the OpLog) : Unlike in Relational mode, there is no filtering of data fields through debezium. The codepath to parse this data into an airbyte type is similar to the initial snapshot case.
Formulate a query or API call to retrieve all fields in document.