framed-data / overseer

Overseer is a library for building and running data pipelines in Clojure.
Eclipse Public License 1.0
97 stars 10 forks source link

Add separable store backends, support SQLite #106

Closed nathell closed 11 months ago

nathell commented 1 year ago

Rationale

I want to use Overseer in a project that already heavily uses SQLite, so I thought I’d use the JDBC store, pointing it to a SQLite table. Also, because I won’t be using Datomic, MySQL, or H2, I wanted to exclude the unused libraries, specifying my Overseer dependency as:

io.framed/overseer {:mvn/version "0.8.9"
                    :exclusions  [com.datomic/datomic-free
                                  com.h2database/h2
                                  mysql/mysql-connector-java]}

However, I ran into two problems:

This PR rectifies both issues. Read individual commit descriptions to see how.

How I checked it

(ns myapp.core
  (:require [overseer.api :as overseer]))

(def job-graph
  {:start []
   :result1 [:start]
   :result2 [:start]
   :finish [:result1 :result2]})

(def job-handlers
  {:start (fn [job] (println "start"))
   :result1 (fn [job] (println "result1"))
   :result2 (fn [job] (println "result2"))
   :finish (fn [job] (println "finish"))})

(def config {:store {:adapter "sqlite",
                     :config "jdbc:sqlite:/tmp/store.sqlite"}})

(def store (overseer/store config))

(overseer/start config store job-handlers) ; run the worker
(overseer/install store) ; set up store
(overseer/transact-graph store (overseer/job-graph job-graph)) ; fire up the job graph

Overseer successfully set up the SQLite store, created the jobs, and ran them to completion.

What this PR doesn’t do

Now that Overseer is able to work with the chosen storage backend’s dependency only, I believe it makes sense to ask users to explicitly depend on that, rather than specify exclusions (it’s easier to add one dependency than to exclude three). In fact, the quickstart already makes such an ask.

However, I’m still not 100% convinced, and so this PR keeps the approach of including all deps.