search

oss-know / airflow-jobs

Apache License 2.0
6 stars 19 forks source link

Add DAG to daily sync github profile from remote clickhouse service #193

Closed crystaldust closed 1 year ago

crystaldust commented 1 year ago

When resolving issue #189 , we found that the github profile can be stored with new engine that will remove duplication, by keeping (github id, updated_at) fields tuple distinct. Which indeed defines a snap shot of a profile at a particular time.

So after changing the new engine, it is possible to sync github profile from remote clickhouse serivce by compare the search_key__updated_at and the new engine will keep removing duplication.

crystaldust commented 1 year ago

Duplication with #194 closing