zerodha / dungbeetle

A distributed job server built specifically for queuing and executing heavy SQL read jobs asynchronously. Separate out reporting layer from apps. MySQL, Postgres, ClickHouse.
MIT License
541 stars 77 forks source link

First aproach to new driver... reusing code... #21

Closed josejuanmontiel closed 1 year ago

josejuanmontiel commented 1 year ago

The idea... include new driver reusing actual code... for later dinamically load by configuration.

With the two (three) actual driver ... use ifs inside actual backend... because new one of new drivers change the code more drastically, decide to implements new backend.

And to reuse actual code in backend that don't change... in this backend include the "parent" of sql backend (the actual one) and only reimplement those methods that didn't change.

The trick its done here

type csvqDBWriter struct { sqlDBWriter }

reusing the actual implementation ... and only change the method ... that realy change...

knadh commented 1 year ago

Hi @josejuanmontiel. Thanks for the inline comments on the PR, but could you please explain here in a few bullet points, what exactly is changing, why, and how it helps. The overall architecture.

josejuanmontiel commented 1 year ago

Hi!

As bonus, need some extra configuration too... i include the comented option to run the jobber_test.go with testcontainer.

TODO:

  1. Adding some free CI ¿circleci?
  2. "Parametrizing" the it test with test container (adding // +build integration)
  3. Adding some test for the new csvq driver
knadh commented 1 year ago

Thanks for the update. Will try to review this some time soon.

knadh commented 1 year ago

Please see this https://github.com/knadh/sql-jobber/pull/23#issuecomment-1730871141