search

onemoredata / bagger

Massive log storage in PostgreSQL
BSD 2-Clause "Simplified" License
9 stars 0 forks source link

Explore rewriting query proxy in golang #59

Closed einhverfr closed 3 months ago

einhverfr commented 8 months ago

Here we are currently writing the query proxy initially in Perl because this matches what we are familiar with and reduces the unknowns in the surface area. However we should explore rewriting the query proxy in golang to see if a) we get better performance, and b) if this helps with other aspects.

einhverfr commented 7 months ago

Initial impressions:

  1. Golang object oriented design and development is a complex and complicated topic. The interplay between interfaces, types, and composition makes the language very difficult to reason about in some of these areas. Additionally OOP in Golang invariable requires a fair bit of metaprogramming and use of reflection. This puts a lot more emphasis on the framework side.
  2. The golang postgres driver is extremely capable and it seems likely that there is a space on this project where it will do extremely well, namely moving
  3. golang interfaces are far less powerful than their Java counterparts
  4. Initial impressions are that once you have framework components down, that development and management of business logic pieces should be fast and easy.
  5. Golang doesn't seem very well suited to business-logic-heavy tasks. I don't see it ever replacing the Perl-based management tooling that we have
  6. Nevertheless, I think Golang would be the strongest choice for the logical replication piece from Postgres to etcd. This would save us a large amount of problems and I think that it is likely that a generic tool could be written and only a big of custom logic needed to do this. This should be an exploration topic for 1.0 since the current LW Agent code has problems due to the nature of the etcd driver.
einhverfr commented 7 months ago

One further note is that of all the Postgres drivers I have evaluated, golang is the only language which has consumers for the pgoutput logical replication plugin. I am considering trying to write a logical replication piece to replace the lwagent for v1 even in golang though I don't think the language is great for most other parts of this stack (at present, opinion could change).