search

DalgoT4D / dbt_stir

This repo is to maintain transformations for SurveyCTO data. We have used Airbyte to pull data from SyrveyCTO. Checkout our Airbyte Repo
0 stars 1 forks source link

Removing Additional columns in final tables #7

Closed vinod-rajasekaran closed 1 year ago

vinod-rajasekaran commented 1 year ago
  1. Hashid columns can be removed from final production tables:
_airbyte_delhi_india__ment_form_2022_hashid TEXT          
  _airbyte_indonesia_stir_bm_2022_hashid TEXT          
  _airbyte_karnataka_stir_bm_2022_hashid TEXT          
  _airbyte_tn_stir_bm_2022_hashid TEXT          
  _airbyte_uganda_bm_2022_hashid

2.Uganda and Indonesia - all districts are not being coalesced and excluded: fields are:

district_acholi TEXT          
  district_bunyoro TEXT          
  district_busoga TEXT          
  district_central TEXT          
  district_karamoja TEXT          
  district_kigezi TEXT          
  district_kota_kediri TEXT          
  district_lango TEXT          
  district_masaka TEXT          
  district_mbale TEXT          
  district_midwestern TEXT          
  district_rwenzori TEXT          
  district_teso TEXT          
  district_westnile

  1. Karnataka form: division_kt does not need to go into final table?

  2. forms_uganda and forms_indonesia can be excluded as they are coalesced into forms

  3. location_indonesia and location_uganda should have been coalesced into region and can be excluded form final table

  4. for sake of easier maintenance create forms_verbose in final merged table rather than across all five intermediate transforms.

Can I review before merge into master and push

vinod-rajasekaran commented 1 year ago

Also add forms_verbose to https://github.com/DevDataPlatform/dbt_stir/blob/main/models/production/classroom_surveys/derived/classroom_surveys_normalized.sql

siddhant3030 commented 1 year ago

Remove district_kota_kediri from indonesia table

siddhant3030 commented 1 year ago

remove location_indonesia and uganda from final table