Improve documentation for capture groups

[boxed]

This conversation on reddit gives some good feedback we should take into account when improving this part of the documentation: http://www.reddit.com/r/Clojure/comments/2xa6f1/specter_data_manipulation_library/

It might be time to restructure the docs generally too. It seems like we've tacked on more and more stuff without refactoring...

[boxed]

ping @crisptrutski @piranha

[piranha]

Yeah, capture groups documentation is a bit confusing - and I still haven't used them. I think that I may use instar more in the near future, so then I might tackle that with understanding. :)

[crisptrutski]

Not my finest documentation :3

Like the idea of introducing the capture operators by simply breaking own examples into

1. What is captured by each group
2. What the path resolves to
3. How the captures are passed to the transform function

We have an example showing resolving-capture and one involving interaction between multiple capture, should also have an example with just non-resolving-capture.

Re: the docs more generally being lifted up, did you have any ideas?

[crisptrutski]

One note on spectre vs instar, is the tree-walking aspect over that it has over instar. This is definitely a super power that I would leverage in my use :)

[boxed]

Hmm.. am I reading this correctly that spectre paths match all the way out into the value of maps. So the instar path [:a :b] is similar to the spectre path [:a :b ALL]?

[crisptrutski]

So the resolving vs. non-resolving dichotomy is more prevalent in Specter.

Here we see vector paths have mostly the same semantics, [ALL :a] is the same as [* :a] (ie. you wouldn't put`[ALL :a ALL])

(update [ALL :a] inc [{:a 1} {:a 2} {:a 4} {:a 3}]) ;; [{:a 2} {:a 3} {:a 5} {:a 4}]

Confusion comes in with predicates being able to apply to the current "value", rather than keys, and that it is "non resolving", so that's probably what confused you:

(update [ALL :a even?] inc [{:a 1} {:a 2} {:a 4} {:a 3}]) ;; [{:a 1} {:a 3} {:a 5} {:a 3}]

This is more general, and means that some of the cases for our #13 and #14 have natural solutions. The filterer wrapper gets closer to Instar behaviour, ie. runs against keys, but still does not resolve (hence ALL is required)

(update [(filterer even?) ALL] inc [1 2 3 4 5]) ;; [1 3 3 5 5]

This adds more power because multiple levels of filtering (or richer things, like tree walking) can be composed, eg:

(update [(filterer even?) LAST] inc [1 2 3 4 5]) ;; [1 2 3 5 5]

To continue the parallels, val-selector and val-selector-one are the same as resolving capture in Instar, except the later one throws exceptions if more than 1 path is resolved.

VAL is like applying capture to the previous terms (so nice for composing).

There's no concept of non-resolving capture.

In terms of our outstanding tasks, #13 and #14 would permit solutions in terms of their StructurePath protocol. The protocol approach is quite interesting, although it does slow things down. The compiled paths then are an interesting feature to turn this around completely.

[crisptrutski]

The walker StructurePath is the big one for me, doing a lot of pattern based transformation in one project.

Consider this contrived example of wanting to clean up user names:

(defn truncate [x] (if (> (count x) 7) (str (.substring x 0 4) "...") x))
(update [(walker :name) :name] truncate
             [{:name "James", :friends [{:name "Really long Name"}]},
              {:name "Another long name", :hobbies "Things with long names"}])
;; What I get
;; [{:name "James", :friends [{:name "Really long Name"}]} 
;;  {:name "Anot...", :hobbies "Things with long names"}]

;; What I'd want
;; [{:name "James", :friends [{:name "Real..."}]} 
;;  {:name "Anot...", :hobbies "Things with long names"}]

(oh, turns out the walking doesn't work the way I'd expect, it doesn't repeat paths when they are deep - I guess because that could create cases when more than one pattern transforms the same value? might take it up with Nathan)

Changing data so that parent does not match lets the child match:

(update [(walker :name) :name] truncate [{:-name "James", :firends [{:name "Really long Name"}]}, {:name "Another long name", :hobbies "Things with long names"}])
[{:-name "James", :firends [{:name "Real..."}]} {:name "Anot...", :hobbies "Things with long names"}]

[boxed]

Interesting. Seems like it should be possible to steal a lot of good ideas from Spectre. Although I guess we'd have to introduce a backwards incompatible 2.0 or a new function (transform2) :(

Leaving that aside though.. do you think a picture like this might help explain the capturing stuff?

[crisptrutski]

Not so sure backwards incompatible changes are needed - a "double splat" ** operator for example could compete with walker. Anyway, back to the docs.

Really like the use of colour - and perhaps animating the appearance of the colours would help even further. What do you think about also including the "resolved path" as an extra line below, so it's clear which terms update that, and how.

[crisptrutski]

Extra lines I guess, expanding to [:users 0] and [:users 1] when the wildcard shows up.

[boxed]

Feedback from reddit:

freshhawk 1 point 11 hours ago 

That use of colour works great, even better than it worked in my head.
as crisptrutski points out, the other part of my initial confusion would be to resolved with some kind of extra lines tracking the path after each step.
So after blue the path is still at the root, after red it's :users, after yellow it's :users * or :users 0,1 (clearly representing the path post-wildcard is a bit tricky).
Definitely on the right track from my perspective.
As an aside, I also really like the idea of a double spat operator doing walking.
I would have commented on github, but I'd rather not link those two online identities, sorry about splitting the conversation.

[crisptrutski]

Lets go for it :) How do you feel about animating? Happy to help out with labour there

[boxed]

I'm open to it... I don't have a clear idea of how that would look, so please give it a shot if you do!

It might look super weird to suddenly have a gif animation in the README or whatever but if it helps explain it's probably worth it :P

[crisptrutski]

Underestimated how much I'd suck at making a GIF, but here's the idea at least

(Couldn't even get Gimp's text to look anything like the screenshot >_<)

Note that I left off highlighting the terms in the body of the function, as that doesn't really interact with the capture mechanism.

Sorry for letting this potato cool!

[boxed]

I don't like the GIF... I mean, I like the frames but it seems like it must be possible to step through them one by one, the animation just makes it stressful to see all the data.

[crisptrutski]

Definitely see how it could be stressful - the first time through you're trying to understand what all those labels mean and associate the colours together. Perhaps it could be improved with better layout and timing, or maybe it's just too dense for an automated animation. I'm quite used to seeing GIFs in repos these days (although usually with better execution) for example for explaining more byzantine emacs/vim plugins, but won't push the suggestion any further.

The key problem seems to be the user not being able to take their own pace, so maybe linking out to an interactive example (even a slideshow) is perhaps the answer. Or just showing frame after frame with guiding text between. Or just getting good copy and using a single image (like yours, except I'd drop the colours in the transform function's body)

Your call

[boxed]

Can you try posting that same gif animation at half speed? And maybe instead of "capture: yes"/"capture: no" you can write "capturing!" when it does capture and just white pixels when it's not? (same with "resolve")?

Maybe that will be fine. I find it hard to imagine an animation at half speed :P

[crisptrutski]

@boxed apologies for letting this rot. If there's still steam here I have some time to try GIFinating again, just wanted to gauge interest first.

[boxed]

I still like the idea :P