chaos-genius / chaos_genius

ML powered analytics engine for outlier detection and root cause analysis.
https://www.chaosgenius.io
MIT License
737 stars 83 forks source link

[BUG] Clickhouse datasource not finishing schema scan #1199

Open fstolba opened 1 year ago

fstolba commented 1 year ago

Clickhouse datasource schema sync is never finished / hangs at in progress, even several hours after adding it. This is on a completely fresh docker-compose based installation. The database is not excessively large from my understanding.

I tried triggering the sync again and, judging from the logs, it seems that CG is indeed working on the sync:

chaosgenius-worker-alerts       | {"asctime": "2023-03-30 21:00:08,476", "levelname": "WARNING", "name": "chaos_genius.controllers.data_source_metadata_controller", "message": "Datasource with id: 2 already in Progress, skipping..", "lineno": 133, "funcName": "run_metadata_prefetch", "filename": "data_source_metadata_controller.py"}
chaosgenius-worker-alerts       | [2023-03-30 21:00:08,476: WARNING/ForkPoolWorker-1] Datasource with id: 2 already in Progress, skipping..

Apart from that, I'm not sure what I'm supposed to be looking for, please let me know what infos/logs are relevant for debugging.

Explain the environment

github-actions[bot] commented 1 year ago

Hello there, thanks for opening your first issue. We welcome you to the community!