search

jitsucom / jitsu

Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days
https://jitsu.com
MIT License
4.13k stars 295 forks source link

Missing primary_key_fields configuration in destination config #1126

Closed ardabeyazoglu closed 2 months ago

ardabeyazoglu commented 2 months ago

Summary

I want to switch to a new postgres table layout with my custom primary key, but i cant change primary key fields from configurator. Editing yaml files dont help either because redis is the meta store.

Adding the configuration option to ui would solve the issue.

Context: This is a big issue with classic version, as it is impossible to partition tables without manipulating the default primary key. I need to add timestamp to the key and partition by month. For clickhouse it may not be a problem, but for postgres/mysql this is very important for big data. For me, it is roughly 2 million events per day and i need to partition and prune old data.

System configuration and versions

I am using jitsu classic, configurator image 1.44.8.

vklimontovich commented 2 months ago

It's unlikely we'll fix it in Jitsu Classic. We are very much focused on Jitsu 2.0. So I suggest you to migrate.

Meanwhile, I suggest you to config the data right in Redis.

ardabeyazoglu commented 2 months ago

Meanwhile, I suggest you to config the data right in Redis.

I thought of this first but configurator will override it each time the form is used, so it is very error prone.

It's unlikely we'll fix it in Jitsu Classic. We are very much focused on Jitsu 2.0. So I suggest you to migrate.

This was on the plan for long term, but we needed quick solution for now. I could not find any migration assistant or any documentation about key configuration differences. Do you have any ?