oss-know / airflow-jobs

Apache License 2.0
6 stars 19 forks source link

Catch ClickHouse DBException when calculating developer role network. #221

Closed crystaldust closed 1 year ago

crystaldust commented 1 year ago

Caculating the developer role network metrics(2 developers modifying the same file) by clickhouse SQL in distributed manner greatly improves the performance, while under complex situations the memory usage would explode, Catch the ClickHouse DBException and skip it, we will lose some data but the whole process keeps moving.