Getting a (sub)schema through a function call (or: how do I extend schemata?)

[lvh]

My application has "steps": atomic behaviors hat it can do (e.g.: make an HTTP request, wait x milliseconds...). They're described by maps with :type keys, the value for which is the unique name for that kind of step. Step implementations have two parts:

1. the schema for the step (since you can pass totally different arguments to a thing that makes an HTTP request than to something that waits x milliseconds)
2. the actual function that takes that map and executes the behavior it describes.

Since it's dispatching based on a value in a map, the easiest way to do this in Clojure appears to be multimethods, at least for the handler portion. I don't know how to do something like this for schemata. (Ideally, of course, preserving a useful error message.)

Since the two components above only make sense when both of them are implemented, I'm probably going to end up writing a macro like this:

(defstep :delay
  {:amount (s/pred #(< 100 % 1000))}
  (fn [request] nil))

... since that macro is free to do whatever it wants, I'm totally open to whatever it is we can do for this :)

[w01fe]

Thanks for the details -- but I'm not sure I understand what you want out of Schema from this. Can you please be a bit more concrete on what your data looks like, and what you want out of it (example ideal usage)?

Not sure if this is relevant, but fns defined with schema have their schemas attached as data for runtime introspection:

user> (s/defn foo :- Long [a :- s/Int b :- String])
user> (s/fn-schema foo)
(=> java.lang.Long Int java.lang.String)

[nahuel]

we discussed this in the clojure ml, I think @lvh touched an uncovered corner case, check this:

https://groups.google.com/d/msg/clojure/99srs8c9eAk/Gt5GhZ8xQQoJ

I think @lvh problem is he needs to define an schema that resolves to another one in runtime. You can do something like that by using s/conditional but he needs a more generic way where you supply a function that returns the final schema based on the validated value. Something like:

(def my-schemas {:http  {:url s/Str}
                 :delay {:seconds s/Int}})
(defn resolve-schema [validated-value] 
  (get my-schemas (:type validated-value)))
(def super-schema 
  (s/resolve-schema-by-value my-fun))    ;; a generic s/conditional

(s/validate super-schema {:type :http :url "http://www.google.com"})

[nahuel]

Note, s/conditional doesn't work for this case because the schemas aren't known in advance, in the @lvh case resolve-schema can be a multimethod extended at any time:

(defmulti get-schema :type)

(def SuperSchema
  (s/resolve-schema-by-value get-schema))

;; you use SuperSchema
(defn some-fun :- SuperSchema  ....)

;; now users can add more schemas at any time, not possible with s/conditional
(defmethod get-schema :http  [v] {:url  s/Str})
(defmethod get-schema :delay [v] {:seconds s/Int})

[w01fe]

I see, the context from the ML is helpful.

From a performance standpoint, doing things this way would definitely be suboptimal.

Internally, we're using something called an abstract-map-schema for this. It basically emulates what you'd expect for an abstract class with one layer of final subclasses, dispatched on a key of your choice (typically :type). Sound like what you want? You can see a gist of the test here:

https://gist.github.com/w01fe/a39fe486cf011be51b2c

I'm happy to provide a gist of the source if this seems useful. I'm not sure I'm ready to add to schema core yet though, since it still feels a little bit clunky.

Getting back to your original problem of looking up a schema directly: I think it wouldn't be too hard to extend the gist @nahuel provided on the mailing list to something that works. But I don't really love the idea, both for performance reasons and because it makes the schemas less declarative and interpretable, although I guess something in this direction is needed if you want a truly open set. I'll have to think about it more, and ideas are also welcome :).

[lvh]

@nahuel has done an excellent job of communicating what I want :-) Thanks!

abstract-map does look an awful lot like what I was asking for, but I'm beginning to think that perhaps it would be more appropriate to build the schema based on a concrete list of supported handlers; that way it'd be possible to configure which handlers you want to support.

I didn't actually know about methods. Pretty useful for what I want to do. Although IIUC, something has to require the module that the defmethod lives in for it to show up; it does not magically show up at compile time.

[w01fe]

On Tue, Aug 26, 2014 at 11:18 AM, lvh notifications@github.com wrote:

@nahuel https://github.com/nahuel has done an excellent job of communicating what I want :-) Thanks!

abstract-map does look an awful lot like what I was asking for, but I'm beginning to think that perhaps it would be more appropriate to build the schema based on a concrete list of supported handlers; that way it'd be possible to configure which handlers you want to support.

Cool, just let me know if you'd like me to share the code.

I didn't actually know about methods. Pretty useful for what I want to do. Although IIUC, something has to require the module that the defmethod lives in for it to show up; it does not magically show up at compile time.

Yes, that's correct.

Just let me know where you end up, and we'll see if there's anything we can do on our end to help.

[lvh]

I'd love the code for abstract-map, yes :) Would it be okay to contribute it to an existing schema contrib project?

[lvh]

For some reason, the code to generate that schema from all the get-schema implementations also doesn't quite work:

icecap.schema> (flatten (for [type (keys (methods get-schema))]
                    [#(= (:type %) type)
                     (get-schema {:type type})]))
(#<schema$eval14451$iter__14452__14456$fn__14457$fn__14462 icecap.schema$eval14451$iter__14452__14456$fn__14457$fn__14462@1146df07>
 {:type :http,
  :url
  {:p? #<core$uri_QMARK_ schema_contrib.core$uri_QMARK_@46330f27>,
   :pred-name URI}}
 #<schema$eval14451$iter__14452__14456$fn__14457$fn__14462 icecap.schema$eval14451$iter__14452__14456$fn__14457$fn__14462@322cbbb6>
 {:type :delay,
  :amount
  {:p?
   #<number$between$fn__14102 integrity.number$between$fn__14102@7770dab0>,
   :pred-name (integrity.number/between 0 60)}})
icecap.schema> (apply s/conditional
         (flatten (for [type (keys (methods get-schema))]
                    [#(= (:type %) type)
                     (get-schema {:type type})])))
{:preds-and-schemas
 ([#<schema$eval14472$iter__14473__14477$fn__14478$fn__14483 icecap.schema$eval14472$iter__14473__14477$fn__14478$fn__14483@51c26a18>
   {:type :http,
    :url
    {:p? #<core$uri_QMARK_ schema_contrib.core$uri_QMARK_@46330f27>,
     :pred-name URI}}]
  [#<schema$eval14472$iter__14473__14477$fn__14478$fn__14483 icecap.schema$eval14472$iter__14473__14477$fn__14478$fn__14483@59ad6037>
   {:type :delay,
    :amount
    {:p?
     #<number$between$fn__14102 integrity.number$between$fn__14102@c76c13>,
     :pred-name (integrity.number/between 0 60)}}])}
IllegalArgumentException No implementation of method: :explain of protocol: #'schema.core/Schema found for class: clojure.lang.Keyword  clojure.core/-cache-protocol-fn (core_deftype.clj:544)

Ostensibly, it did actually create the schema, but any attempt to display it (e.g. just the expression Step) causes that exception.

[lvh]

I'm also not to sure how to write the function that produces Plan/Schema. Plan is recursive, and recursive seems to require a (quoted) named reference, so the "obvious" thing was a nested let, which throws at compile time since the Plan reference inside recursive can't be resolved.

(defn schema
  "Builds the schema for a plan that allows the given step types."
  [step-types]
  (let [step-preds-and-schemas (for [type step-types]
                                 [#(= (:type %) type)
                                  (get-schema {:type type})])
        Step (apply s/conditional
                    (flatten step-preds-and-schemas))
        Plan (let [Plan (s/recursive #'Plan)]
               (s/conditional
                map? Step
                vector? (with-one-or-more [Plan])
                set? (with-one-or-more #{Plan})))]
    Plan))

So I guess I'm stuck with abstract-map? :)

[nahuel]

@lvh: seems like your get-schema is returning an invalid schema, probably you are doing:

(defmethod get-schema :http  [v] {:type  :http ...})   ;; wrong

instead of:

(defmethod get-schema :http  [v] {:type  (s/eq :http) ...})   ;; good

[w01fe]

The code for abstract-map-schema is here:

https://gist.github.com/w01fe/a39fe486cf011be51b2c

Consider it under EPL; feel free to contribute to a schema-extension project for now.

Like @nahuel says, it looks like your schema is incorrect. Maybe best to test them individually before trying to bring all the pieces together?

As far as recursive schemas, to construct cycles in Clojure you need a reference of some type, which is what recursive expects. You can do this with def by referring to the var in its own expansion, or you can use an atom like so:

(def church (let [r (atom nil)] (reset! r (s/maybe {:n (s/recursive r)}))))
(s/validate church {:n {:n {:n nil}}})

[lvh]

Thanks, @w01fe and @nahuel! This fixes my issue :) I'll try to contribute abstract-map to sfx/schema.contrib.