Status-website: unstringify the label in gh_issues endpoint

[apentori]

What to do

1. Extract the labels into another psql table with dbt models
2. Create a relationship between the two tables (gh_issue and gh_issue_label)

We need to add another Hasura API call during the init to create the relationship between table: https://hasura.io/docs/latest/schema/postgres/table-relationships/create/#step-2-create-an-object-relationship

[apentori]

DBT model modification:

• https://github.com/status-im/dbt-models/commit/bc4660752241cfdc97b27fbec54eff2ad6a44b03
• https://github.com/status-im/dbt-models/commit/c1ef91993a24e28a0863d0419c6c07617124177e

Hasura init modification:

• https://github.com/status-im/infra-bi/pull/111

[apentori]

There is an issue when trying to automate the relation between the two endpoint.

It appear there is 21561 issues_labels without any related issue in the db. That block to automate the relation between the two endpoint

[apentori]

Unmatching labels with issue fixed by cleaning all the data.

Each airbyte sync remove the primary and secondary key from Hasura db. It doesn't seem to be problematic at first sight, but it will surely become an issue later.

When checking the sync, some constraint might be syncable:

To add constraint in dbt: https://docs.getdbt.com/reference/resource-properties/constraints

Going to check if it's possible to specify the schema of only some table

[apentori]

Constrains are not available in our current dbt version ....

Unfortunately, this feature is not available in dbt Core version 1.4

The relation between object seems to work even if Airbyte remove the postgres constraint.

[apentori]

In order to keep constraint on data, i had to add a dbt transformation on the hasura database:

Modification implied

• Airbyte write data in the schema airbyte in the Hasura db.
• A new dbt profile is set to connect to the hasura db and replicate the data from one schema to the other (airbyte -> public) with the constraint added.
• New dbt model for the constraints.
• Update to the airflow dag to add the second dbt run.

But why ?

• why not do it in Airbyte ; Airbyte doesn't seems to replicate constraint across database. When trying to run dbt model directly in the Airbyte sync, it didn't find the correct tables.
• Do we need to add constraint at each run of the dbt models ? Yes, the airbyte sync remove the constraint.
• Why not apply the first dbt transformation, with constraints, directly on Hasura ?
  • Other BI stuff need the github transformed data, pushing the raw data direclty to hasura and doing the dbt transformation there is not good.
  • DBT doesn't allow multiple postgres db connection: impossible to run the models twice with different db destinations
  • Transferring the raw github data to hasura and doing twice the dbt transformation would imply sending more data than necessary.
  • Duplicating the flow toward both db (hasura and main db) would break the API limit