search

DalgoT4D / dbt_lahi

0 stars 2 forks source link

upgrade LAHI's dbt #49

Open fatchat opened 4 months ago

fatchat commented 4 months ago

Tech4Dev is upgrading all Dalgo clients to Airbyte's Destinations v2 feature. As part of this upgrade, all ingested data will be extracted from _airbyte_data according to the detected schema.

The dbt code needs to be updated to be able to read from these new tables; this is the motivation behind this GitHub issue.

For KoboToolbox, the column _airbyte_data current holds JSON having the schema {data, _id, endtime, end, _submission_time}, and we use dbt to extract these into separate SQL columns.

After the migration to Destinations v2, Airbyte will do this first step of flattening. Our dbt code will therefore need to read from these columns rather than from _airbyte_data

The column _airbyte_ab_id has been replaced by _airbyte_raw_id, and the column _airbyte_emitted_at will become _airbyte_extracted_at. These two columns will appear in the extracted tables along with the columns from the source schema

Connector normalization has already been deprecated as part of this upgrade, since this step is being done as part of the ingest stage.

fatchat commented 4 months ago

May also need to update sources.yml