Support "calculated key" crumbs functions in addition to predicates

[crisptrutski]

@boxed maybe I've added enough complexity already, but here's another proposal :P

Consider a dummy case like (transform {:a [4 3 1]} [:a last] inc).

Obviously that won't work, as it'll be called on the indices to return true/false. But the application of a function which takes existing keys/indices, and returns keys/indices to operate on would be powerful for getting/transforming sorted data portions (just get the last 5 messages for every channel on this org) or creating new keys based on existing data (say, generating browser prefixes in css dsl).

Not sure what the syntax should look like, maybe introduce another symbol like for capture, like [:a (%^ last) :b] - no appropriate symbol springs to mind. There's a more natural one for predicates: [:a (%? odd) :b].

The most extreme proposal would be to have the behaviour based upon whether the function name ends in a question mark, with the syntax above explicit in either case to override convention.

Felt a bit of pain with some code written today where this would have rocked :)

[crisptrutski]

What may not be clear above is that we'd want to be able to produce single keys or lists of keys, from the existing keys. Maybe two different wrapper symbols for these cases. And maybe take optional params for some partial application.

[boxed]

Can your example use cases be handled by a 2-arity predicate function? That is, a predicate that gets passed not just the key but the object that contains the keys. So your "last" case would be handled by a predicate #(= (- 1 %1) (length %2))). Or something?

[leontalbot]

@crisptrutski +1. I would use those (at least the predicate one) I suggest [:a (%# last) :b] and [:a (%? odd) :b]

@boxed good idea!

[crisptrutski]

Arity 2 works for the main case - selecting keys based on entire set. Passing object rather than keys is interesting, gives extra power I hadn't considered.

Could provide combinators and/or common functions, so users needn't figure out wrappers like in your example.

May give surprises when used with native javascript functions, but we can slap on a warning and note on using .bind to solve work around it.

The other case, of generating new keys not in the object, is not covered. OK I guess, we have capture to help in those scenarios.

[boxed]

I'm just spitballing here... but what about meta data on the functions? So:

(transform {}
  [:foo ^generate-keys #([:new :keys :here])])

and

(transform {}
  [:foo ^key-and-object #(= (- 1 %1) (length %2))))

or something. Those names are horrible, but it feels like these cases could be unusual enough to have more verbose syntax.

[crisptrutski]

The trick with metadata in the path does not seem to work with vars

user=> (meta ^:generator #())
;; {:generator true}
user=> (meta ^:generator last)
;; nil

We could use (with-meta last {:generator true}), but then getting really verbose.

Established a convention here for wrapping values, think that'd be more consistent (even if we give them longer names).

Like generator vs predicate as our nomenclature, and happy with just passing object as second param for arity-2 predicates, and keeping predicates as default.

[crisptrutski]

Examples:

(defn last? [i xs] (= i (dec (count xs))))

=> (transform {:list [1 2 3]} [:list last?] inc)
;; {:list [1 2 4]}

(defn prefix [key] (keyword (str "pre-" (name key))))
(defn prefix-all [keys] (map prefix keys))

=> (transform {:a 1, :b 2} [(generator prefix-all)] :dummy)
;; {:a 1, :b 2, :pre-a :dummy, :pre-b :dummy}

There's one really bad thing about generators though.. signature of [key] -> [key] doesn't help much for providing values to the transformation.

Maybe [key] -> {key -> [key]} is better..

[crisptrutski]

With changed generator contract:

(defn prefix [key] (keyword (str "pre-" (name key))))
(defn prefix-all [keys] (zipmap keys (map prefix keys)))

=> (transform {:a 1, :b 2} [(generator prefix-all)] dec)
;; {:a 1, :b 2, :pre-a 0, :pre-b 1}

[boxed]

I've let this stew a few days but my brain still won't parse those suggested examples, so I don't think they're a super good fit...

While I was trying to write this comment, it occurred to me that we have a piece of the syntax we can exploit more. Our paths are always [:foo :bar :baz], but what if that's just the syntax for literal paths, while you can in fact have functions there to return a list of paths instead? In a way that would mean we could think of (transform foo [*] inc) as being syntactic sugar for (transform foo #(expand-path [*]) inc or something.

I guess that would mean your example above would be something like:

(def input {:a 1, :b 2})

=> (prefix-keys input)
;; [[:a] [:b]]
=> (transform input
          prefix-keys dec)
;; {:a 1, :b 2, :pre-a 0, :pre-b 1}

I'm too tired/confused to write prefix-keys properly in clojure right now, but in python it'd be basically [['pre-' + key] for key in input.keys()].

What do you guys think?

[crisptrutski]

Appreciate your concerns about complexity - starting to walk a fine line between expressiveness and obfuscation. Maybe we're overloading this function to much and should just abandon this and #13 :dolphin:

Looking at your suggested alternate, I have two objections:

1. The prefix-keys function does not provide information about which old path to provide values for the new path. (prefix-keys input) => {:a [:pre-a], :b [:pre-b]} could resolve ambiguity. Could support association lists too, as that'd simplify function definitions ((for [k (keys input)] [k [(keyword (str "pre-" (name k)))]) for this one)
2. Replacing the vector path is less composable and breaks symmetry too much for me - imagine ending up having to use function composition to get back the feature - eg. the case to run the prefix-key style function as a middle segment ([:a :b prefix-keys :c :d])

Would end up with ad-hoc things like below (or abstracting it to a high order function)

(defn prefix-keys-deep [pre-keys post-keys] 
  (fn [input] (concat pre (comp prefix-keys #(get-in % pre-keys)) post)))

Feel free to just kill issues!

[boxed]

1. Oops, yea you're right.
2. Hmm yea.. that's a bit fugly.

I still have a strong feeling there's something pretty nice lurking in the shadows somewhere :P

Maybe to get something nice and composable we'd have to go back and question the basic vector path stuff though, and that seems like a big loss :(

[crisptrutski]

I still like [:a :b (predicate odd?) :c :d (generator prefix-with "pre-") :e :f] as syntax. Support metadata on functions as secondary sugar for power users. :lemon:

[crisptrutski]

RE: the case where "what maps to what", we could support a special case where a seq of exactly the same length as input is returned, and assume it maps the keys 1-1 in order.

[boxed]

Funny, https://github.com/nathanmarz/specter does exactly the "last" feature this ticket describes.

[crisptrutski]

Thanks for the link - interesting library, quite a different approach. Very new library from such an old name :)