CodeForPhilly / cfp-data-pipeline

7 stars 3 forks source link

Put github and slack into separate schemas for #27 #29

Closed dherbst closed 3 years ago

dherbst commented 3 years ago

For #27 puts tap_github into views_github and tap-slack into views_slack

For example:

datawarehouse=# \dt views_github.*
                 List of relations
    Schema    |       Name       | Type  |  Owner   
--------------+------------------+-------+----------
 views_github | assignees        | table | postgres
 views_github | collaborators    | table | postgres
 views_github | contributors     | table | postgres
 views_github | issue_labels     | table | postgres
 views_github | languages        | table | postgres
 views_github | repository       | table | postgres
 views_github | stargazers       | table | postgres
 views_github | team_members     | table | postgres
 views_github | team_memberships | table | postgres
 views_github | teams            | table | postgres
(10 rows)
chriscardillo commented 3 years ago

Implementation looks good, though #28 suggests that the work of creating schemas should be moved to its own DAG.

If #20 is merged in, it should be as easy as moving datawarehouse_setup.py into its own DAG folder.

dherbst commented 3 years ago

Sounds good.

dherbst commented 3 years ago

Here are the schemas the data will go into:

datawarehouse=# \dn
    List of schemas
    Name    |  Owner   
------------+----------
 public     | postgres
 tap_github | postgres
 tap_slack  | postgres
 views      | postgres
(4 rows)