a way to get a map of functions

[yogthos]

I'm currently using Yesql in Luminus, but I'd like to either switch to HugSQL or support it as an alternative. Currently, I manage the connection using the conman wrapper library I wrote.

One of the things I wanted to do was to ensure that the transactional connection was used by default. I find that the behavior of having to remember to pass it in is quite error prone, and you only get an error in case the transaction would fail.

My approach was to use a dynamic var store the connection, and then rebind it within transactions. What I ended up doing with Yesql is creating a shadow namespace where it declares its functions and then wrapping those with functions I generate. he functions defined by conman use the dynamic var and get the transactional connection by default.

It would be really nice to have the ability to get a map of anonymous functions keyed on their names from HugSQL that could then be used to declare my own functions in conman.

If something like this was available, that would be great :)

(defn parse-queries [file]
  (reduce
    (fn [fns {{:keys [name doc]} :hdr sql :sql}]
      (assoc fns
        (-> name first keyword)
        {:doc doc
         :fn (sql->fn sql)})
      {} (parsed-defs-from-file file))))

[csummers]

I think I'm open to adding a function or macro to accomplish this. My personal preference on database-calling functions is to pass in the db-spec/connection/pool/transaction object as the first parameter. However, the fact that quite a few folks may be using Luminus/conman and would be familiar with its usage pattern definitely lends some weight to this. Let me sleep on it! I'm marking this as a future enhancement.

[yogthos]

Sounds great. The way I set it up in conman is that you can either use the connection value that the functions have been bound to or pass a connection explicitly. Exposing a map of functions to the user opens up a lot of possibilities. For example, you could do things like:

(def queries (parse-queries "queires.sql"))

(defn get-users [conn query-params]
  ((get-in queries [:get-users :fn]) conn query-params))

This gives the users the option to do things explicitly as well as having the library define the functions for them.

[csummers]

@yogthos I slept on it, then added some macros for this:

• map-of-db-fns
• map-of-sqlvec-fns
• map-of-db-fns-from-string
• map-of-sqlvec-fns-from-string

These functions return a hashmap of the form:

{:fn1-name {:meta {:doc \"doc string\"} :fn <fn1>}
 :fn2-name {:meta {:doc \"doc string\" :private true} :fn <fn2>}}

I've started a 0.4.1-SNAPSHOT in master. Would you be willing to clone the repo and do a local install to ensure this will work for your needs?

Since hugsql is a meta-project, you can use the alias lein install-all to install the sub-projects locally in the correct order.

[csummers]

Also, something to consider when wrapping up functions is the original arities of the functions produced by HugSQL. From the docs:

The functions defined by def-db-fns have the following arities:

  [db]
  [db param-data]
  [db param-data options & command-options]

where:

• db is a db-spec, a connection, a connection pool, or a transaction object
• param-data is a hashmap of parameter data where the keys match parameter placeholder names in your SQL
• options is a hashmap of HugSQL-specific options (e.g., :quoting and :adapter)
• & command-options is a variable number of options to be passed down into the underlying adapter and database library functions.

[yogthos]

Great, I can definitely help with test driving this. Reading the code, I don't think the current approach will quite work for my use case though. The use case I have is that I wish to avoid implicitly defining the functions by HugSQL and have the map-of-db-fns return them as a map explicitly. That way I can define my own wrappers using the names of the keys in the map. Unless I'm mistaken, def-expr will already define the functions before that though.

I propose splitting out the code in def-expr that compiles the body of the function, so then it could either be used to define the function or create an anonymous function that you could then stick in the map.

Let me know if I'm making sense here. :)

[csummers]

def-expr is only for the clojure expression support, and expressions are handled completely internally by the hugsql-generated functions.

In the just-committed code with map-of-db-fns, you're given a map of the function name (as a keyword), and the value is a hashmap of :meta and :fn. The function is an anonymous function, so at this point we haven't def'd anything. You should be able to create wrappers around the anonymous function.

Does that make sense?

[yogthos]

Ah ok I think so, I misread the last bit of def-expr, but I see it just returns an anonymous function that can be used to intern the symbol. I'll play with this today, but it looks like this should be perfect.

[csummers]

Great! Let me know if you run into any issues. I'm happy to answer any questions to get this right.

[csummers]

Just for clarification on expression compilation: All of the def-*-fns, map-of-*-fns make sure that any encountered clojure expressions in the SQL are compiled before creating any user-facing functions (either w/ a def or a hashmap+anonymous functions). This is the doseq calling compile-exprs. However, expressions are internally handled and behind-the-scenes. You shouldn't need direct access to them.

map-of-db-fns uses db-fn-map, which uses db-fn* to give you the anonymous function you're seeking.

[yogthos]

I've got the latest compiled locally, and so far everything's working exactly as expected. Thanks on the quick turnaround on this by the way.

[csummers]

So I thought of an issue you might run into and I think I have a fix for it. Since conman will be acting in a "supervisory" role to define hugsql functions, you'll have to account for the difference between a database function and a snippet function. Snippet functions do not take database connections, so you will not want to define them as such. A snippet function's arities are:

[sql]
[sql options]

So, in the hashmap you receive from map-of-db-fns, snippet functions are included, but until commit c20b381 you didn't have a way to know which functions are snippets. Snippet functions now have :snip? true metadata in order for you to handle this case.

Does that make sense?

[yogthos]

Ah yes, good catch, this way I'll can simply filter on that.

[yogthos]

So, this seems to be working as expected, does it look good to you?

(defn load-queries [filenames]
  (reduce
    (fn [queries file]
      (->> (hugsql/map-of-db-fns file)
           (remove #(get-in % [:meta :snip?]))
           (into queries)))
    {}
    filenames))

(defmacro bind-connection [conn & filenames]
  `(let [queries# (load-queries '~filenames)]
     (doseq [[id# {fn# :fn}] queries#]
       (intern *ns* (symbol (name id#))
               (fn
                 ([] (fn# ~conn {}))
                 ([params#] (fn# ~conn params#))
                 ([conn# params#] (fn# conn# params#))
                 ([conn# params# opts# & command-opts#]
                  (apply fn# conn# params# opts# command-opts#)))))))

[csummers]

I think users would expect the snippets to actually be defined also, since they would be using the snippets in the same namespace.

[yogthos]

Ah right, so perhaps something like this:

(defn load-queries [filenames]
  (reduce
    (fn [queries file]
      (let [{snips true
             fns   false}
            (group-by
              #(-> % second :snip? boolean)
              (hugsql/map-of-db-fns file))]
        (-> queries
          (update :snips into snips)
          (update :fns into fns))))
    {}
    filenames))

(defmacro bind-connection [conn & filenames]
  `(let [{snips# :snips fns#   :fns} (load-queries '~filenames)]
     (doseq [[id# {fn# :fn}] snips#]
       (intern *ns* (symbol (name id#)) fn#))
     (doseq [[id# {fn# :fn}] fns#]
       (intern *ns* (symbol (name id#))
               (fn
                 ([] (fn# ~conn {}))
                 ([params#] (fn# ~conn params#))
                 ([conn# params#] (fn# conn# params#))
                 ([conn# params# opts# & command-opts#]
                  (apply fn# conn# params# opts# command-opts#)))))))

[csummers]

Yes, something like that will get users a consistent experience across both conman and hugsql.

If you're fairly confident that these additions to hugsql will work, I can push out a 0.4.1 release a bit later today. Let me know. Thanks!

[yogthos]

Thanks, so far I haven't run into any problems. I think it's probably safe to push this out. :)

[csummers]

This is released in 0.4.1

[yogthos]

Awesome! I'm just cleaning up conman, and probably going to push out a new version of it today as well. :)

[yogthos]

probably worth noting this in the Yesql comparison section http://www.hugsql.org/#faq-yesql

[yogthos]

And pushed out new conman that's backed by HugSQL now, I'll have to do some updates for Luminus, mostly in the documentation and then I'll switch it over to use the latest conman.

[csummers]

Very cool!

Since HugSQL is not an exact drop-in for Yesql, you may need some kind of "converting Yesql to HugSQL" doc to aid existing users. I'd be happy to answer any questions along those lines when the time comes.

[yogthos]

Sounds great, I'll probably add this in the changelog for Luminus. The nice part is that for most Yesql queries the transition should be very straight forward.

The timing on this worked out very well for me, I'm just wrapping up the beta for the Clojure web dev book I'm publishing with pragprog, and I'm going to update it to reflect the change. Would've been unfortunate if it got published right before I switched Luminus off Yesql. So, in other news HugSQL will get a bit of promotion there as well. :)

[csummers]

Sounds great! What good timing! I really appreciate all of the work you've put forward in the Clojure web development space in libraries, documentation, and the book. Great stuff!

[yogthos]

Thanks, I really appreciate it. It's been great seeing the ecosystem grow, and to be able to work with great libraries such as yours.