jitsucom / jitsu

Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days
https://jitsu.com
MIT License
4.08k stars 288 forks source link

Error while loading events, show in logs of one of clickhouse shard #1119

Closed stouch closed 1 month ago

stouch commented 2 months ago

Summary

Got error when loading events, but ingest, rotor and console work :

Capture d’écran 2024-08-20 à 12 16 49

shard1 error:

Capture d’écran 2024-08-20 à 12 18 49

And rest of services look OK :

shard0

Capture d’écran 2024-08-20 à 12 19 15

bulker:

Capture d’écran 2024-08-20 à 12 20 03

ingest:

│ time="2024-08-19 16:02:36" level=info msg="[metrics_server] Starting metrics server on 0.0.0.0:9091"                                                                   │││ │ ││││││  │    │         │           │                 │                                                 
│ time="2024-08-19 16:02:36" level=info msg="Starting http server on 0.0.0.0:3049"                                                                                       │││ │ ││││││  │    │         │           │                 │───────────────────────┐───────────────┐────────┐
│ time="2024-08-19 16:04:01" level=error msg="[ch_events_log] Error preparing batch: code: 60, message: Table default.events_log does not exist"  

System configuration and versions

k8s install of Jitsu v2.8.0

(https://github.com/stafftastic/jitsu-chart)

vklimontovich commented 2 months ago

Seems like the table that for storing event logs is configured incorrectly. It should be created in newjitsu_metrics DB, see compose file for a reference

stouch commented 2 months ago

It might be because of https://github.com/stafftastic/jitsu-chart/blob/fb5d459c9c831d25c99a49981c92de43b174c1d8/templates/_helpers.tpl#L145 -> when we use the jitsu-chart, it uses by default the "default" db.

It changed recently ?

stouch commented 2 months ago

@hendrikheil After investigation, I found that this error of unknown table does not always occur (sometimes the data load well in my console). Then I went to my k8s clickhouse shards and find out that in one of the shard the db does not exist :

I got 1 replica and 2 shards :

This is the shard for which the table does not exist :

Capture d’écran 2024-08-21 à 15 56 43

That shard does have it :

Capture d’écran 2024-08-21 à 15 56 08

Any idea what I could do to fix this ?

github-actions[bot] commented 1 month ago

This issue is stale because it has been open for 30 days with no activity.

github-actions[bot] commented 1 month ago

This issue was closed because it has been inactive for 14 days since being marked as stale.