shayonj / pg_flo

Stream, transform, and route PostgreSQL data in real-time.
https://pgflo.io
Apache License 2.0
625 stars 11 forks source link

可以处理聚合结果吗 #31

Open night1008 opened 4 days ago

night1008 commented 4 days ago

比如把聚合结果转存到其他表

select date, count(1) as from table group by 1

还有可以把多张不同字段名称的表通过字段映射到同一张表吗?

shayonj commented 4 days ago

Hi! Thank for taking the time

可以处理聚合结果吗 如把聚合结果转存到其他表 select date, count(1) as from table group by 1

Unfortunately, not today, however, I have spoken to a few folks about the ability to control in what form they would like the data to be synced, and aggregation is a common one.

I am thinking, users can write the sql and sync + transform + route the data to another table. And manage the sql through a yaml file. Would something like this, help w/ your use case?

还有可以把多张不同字段名称的表通过字段映射到同一张表吗?

I would not recommend, writing to the same table again, because of logical replication and the replication slot is created under the same table. Its better if you route to a different table, using the routing feature.

night1008 commented 1 day ago

Thanks for your response.

  1. Looking forward to the the data sync with aggregation.
  2. I have some ETL requirements, for example, union different tables, yes, it would sync to another table, but maybe the column name or column type is not unified, as follows.
table1 id1(int) name1(string) age1(int)

+

table2 id2(int) name2(string) age2(string)

=>

table3 id3(int) name3(string) age3(int)