The "kiba" binary command will be deprecated

[thbar]

(this is a roadmap notice for Kiba users, as much as a reminder for myself)

In 4 years of production use with Kiba, I've had a lot of time to investigate the various patterns of usage on real cases.

Up to v1.0.0, Kiba's way to run jobs was to use bundle exec kiba my_script.etl, which involves reading and evaluating the .etl file, as seen here:

https://github.com/thbar/kiba/blob/378aca8200064e8d597f2581b818bc21824afcca/lib/kiba/cli.rb#L10-L13

In the mean-time, I've introduced official support for a programmatic API, an API that was initially introduced to allow in-process testing.

The programmatic API allows everything the "command" mode supports, plus much more, and actually encourage better coding practices.

For instance:

• API mode allows to pass live variables (rather than just ENV configuration from command line or JSON configs from files)
• Doing so permits to wrap resources open/close around running a job
• API mode makes it easier to run testing on an ETL process (via minitest/rspec) directly in-process (which allows stubbing/webmock etc), rather than via a command call
• API mode enforces use of clean modules with explicit loading, rather than polluting the top-level namespace with global methods (https://github.com/thbar/kiba/wiki/How-do-you-define-ETL-jobs-with-Kiba%3F)
• API mode allows to run jobs from Sidekiq or background job systems, from an HTTP call (if the job is fast), without necessarily waiting for a command line binary to run - this supports more dynamic interactions (e.g. a job is created in reaction to an external event received via HTTP or a websocket)

For all these reasons, I plan to deprecate the kiba command in an upcoming version (probably v3).

Ideally I'd like to provide a new gem that will bring back that functionality as opt-in, to ensure v3 can be retrofitted easily, but that gem will likely not be supported anymore when we reach v4.

[thbar]

Work has started in branch deprecate-kiba-command-line. This won't be merged until v3 though.

I plan to issue a version like "2.9" which will only display the deprecation warning, and to create an external gem to reimplement that (just as a temporary measure to help migration for legacy scripts).

[thbar]

A gem to provide a kiba-legacy-cli command is available at https://github.com/thbar/kiba-legacy-cli.

[thbar]

I've published v1.0.0 of the kiba-legacy-cli command at https://rubygems.org/gems/kiba-legacy-cli. Will be usable with Kiba v3 (although I recommend dropping the use of the CLI completely, ideally).

[ngrilly]

Reminds me of a Python ELT tool I developed years ago to build data marts, and I made a similar choice :) It was a library and not a binary command, which made it more explicit, less magic, more flexible, easier to test.

[thbar]

v3.0.0 is out, and I rewrote the wiki mostly from scratch to provide updated recommendations. Closing.