[IDEA] TiddlyWiki filter visualizer/helper tool

[saqimtiaz]

A TiddlyWiki filter visualizer/helper tool, like regexper.com and reg101.com are for regular expressions, would be extremely useful.

I was explaining the basics of TiddlyWiki filters to a colleague last night and realized that it felt very reminiscent of explaining regular expressions.

This would be a far from trivial undertaking, but I think the potential dividends make it worthwhile. The challenge as always is finding the available resources.

[Jermolene]

Great idea. It might be interesting to sketch out how we might visualise some example expressions.

[AnthonyMuscio]

Yes, great idea, but without making it a big job just documenting a few values for the regexp and splitregexp that apply to common tiddlywiki use cases would be a good start.

eg; \n \t

But yes, the filter visualizer is a larger project, as Jeremy did in the past the idea of a filter constructor similar to how excel assists the creation of a formulae can be the "end goal".

Mohammad has started the Doco here http://tw-regexp.tiddlyspot.com/#Using%20Regexp%20in%20Tiddlywiki and TT is well versed in this use of regex and endeavoured to do something similar in the past.

Words, sentences (delimited by "." period) and others come to mind as useful without needing to fully grasp regex.

With the new operators https://tiddlywiki.com/prerelease/#search-replace%20Operator we also need to document common values we may set. For example replacing \n with \n\n or vici versa.

Tony

[Jermolene]

I found this example of annotating a regular expression that I thought was pretty effective:

The key thing is pulling the expression out into multiple lines that can be individually annotated. I think one could make a case for flipping it vertically, but it would be interesting to experiment.

It also occurs to me that we might do better given that we're not constrained by monospaced unstyled text.

[saqimtiaz]

@Jermolene that is a good example indeed.

Here is a very quick attempt at doing something similar with a filter:

[Jermolene]

Good stuff @saqimtiaz. I think we might need to include the punctuation characters in the breakdown because they are a major source of confusion.

I had a quick try in wikitext:

|`[tag{$:/MySpecialTag}has<field>get[text]!is[blank]]` | |
|`                                                   ` | |
|`[                                                 ]` |A run of filters that are piped together |
|` tag                                               ` |"tag" operator returns input titles that have a specific tag |
|`    {               }                              ` |Curly braces means that the operand value will be obtained from a tiddler |
|`     $:/MySpecialTag                               ` |Title of the tiddler containing the operand value for the "tag" operator |
|`                     has                           ` |"has" operator returns input titles that have a specific field |
|`                        <     >                    ` |Angle brackets means that the operand value will be obtained from a variable |
|`                         field                     ` |Name of the variable containing the operand value for the "has" operator |
|`                               get                 ` |"get" operator returns a specific field of the tiddlers identified by the input titles |
|`                                  [    ]           ` |Square brackets means that the operand value is given literally |
|`                                   text            ` |Name of the field to be retrieved by the "get" operator |
|`                                        !          ` |Bang means that the following operand should be negated |
|`                                         is        ` |"is" operator returns input titles that meet a specific criteria |
|`                                           [     ] ` |Square brackets means that the operand value is given literally |
|`                                            blank  ` |"blank" filters titles that are empty  |

[saqimtiaz]

@Jermolene that is terrific, thank you. As you say we can do much more with the presentation too, making it interactive, tying in documentation etc.

In an ideal world we could re-use our existing filter parser but it would have to be done without a performance cost for filter compilation.

Edit: I do believe the existing parser does most of what we need except giving us start/end ranges, which we should be able to work around.

[saqimtiaz]

Executing in a browser console for a TiddlyWiki tab: JSON.stringify($tw.wiki.parseFilter("[tag{$:/MySpecialTag}has<field>get[text]!is[blank]]"),undefined,4);

We get:

[
    {
        "prefix": "",
        "operators": [
            {
                "operator": "tag",
                "indirect": true,
                "operand": "$:/MySpecialTag"
            },
            {
                "operator": "has",
                "variable": true,
                "operand": "field"
            },
            {
                "operator": "get",
                "operand": "text"
            },
            {
                "prefix": "!",
                "operator": "is",
                "operand": "blank"
            }
        ]
    }
]

[Jermolene]

Snap @saqimtiaz I was about the post the same thing.

[saqimtiaz]

Just occurred to me that the filter parse tree looks a bit different in 5.1.23 pre-release with the support for multiple operands:

[
    {
        "prefix": "",
        "operators": [
            {
                "operator": "tag",
                "operands": [
                    {
                        "indirect": true,
                        "text": "$:/MySpecialTag"
                    }
                ]
            },
            {
                "operator": "has",
                "operands": [
                    {
                        "variable": true,
                        "text": "field"
                    }
                ]
            },
            {
                "operator": "get",
                "operands": [
                    {
                        "text": "text"
                    }
                ]
            },
            {
                "prefix": "!",
                "operator": "is",
                "operands": [
                    {
                        "text": "blank"
                    }
                ]
            }
        ]
    }
]

[saqimtiaz]

I worked on this a bit late last year and one of the things that would make this easier in terms of mapping the output of the parser back to the original filter string, is adding start and end ranges to the filter parser.

Any thoughts on whether there would be performance considerations if we did so?

[Jermolene]

I worked on this a bit late last year and one of the things that would make this easier in terms of mapping the output of the parser back to the original filter string, is adding start and end ranges to the filter parser.

Any thoughts on whether there would be performance considerations if we did so?

I don't think it would have a huge impact, given the way that we compile filters, and I agree that it would be useful.

[pmario]

@Jermolene ... Did you ever think about the possibility to have a node.tree, that also contains some info about the "regexp, that created it". ... So imo it would be possible to re-create the plain text out of the tree.

IMO this would also help here, since it may give us the "punctuation characters" back, which IMO would make creating the documentation easier.

[pmario]

... But adding too much of "debug" logic may be a performance hit.

[Jermolene]

@Jermolene ... Did you ever think about the possibility to have a node.tree, that also contains some info about the "regexp, that created it". ... So imo it would be possible to re-create the plain text out of the tree.

IMO this would also help here, since it may give us the "punctuation characters" back, which IMO would make creating the documentation easier.

... But adding too much of "debug" logic may be a performance hit.

Apologies, I don't know exactly what you're suggesting. The "regexp that created it" is confusing.

It is already possible to recreate the text of a filter from its parse tree, but insignificant whitespace is lost.

[AnthonyMuscio]

Since this issue is active again, I defer to others about the coding here, but I would like to add something that applies to many cases in TiddlyWiki development.

It is possible to generalise the solution in someways so users can make use of the same code patterns, even just the odd macro or widget.

Tony

[saqimtiaz]

Posting some details of my work on this so far so as to keep the ball rolling when I have the time to get back to it.

The image below shows the dynamic output of a very quick hack of a widget that tries to re-create the filter syntax description we have created above in wikitext:

The call to the widget is: <$filter-diagram filter="[tag{$:/MySpecialTag}has<field>get[text]is[tiddler]!is[blank]]"/>

For the explanatory text, it is just a case of needing to have better documentation and strings that we can use for the filter operators etc. The catch is of course how do we include that without adding significant bulk to the file size, and we need to make sure it can be translated as well. Less of a concern if we think of this as a plugin, but it would add a lot of value as a core component.

Naturally we can do a lot better for the UI. One concern is that displaying the filter syntax breakdown and explanations side by side does not work in the fixed-fluid story layout (or on mobile). So I am considering moving the explanations below the filter syntax breakdown, where clicking on a part of the syntax highlights the appropriate description and vice versa (Somewhat inspired by http://apps.workflower.fi/vocabs/css/en#selector).

[Jermolene]

The image below shows the dynamic output of a very quick hack of a widget that tries to re-create the filter syntax description we have created above in wikitext:

Excellent, looks great.

For the explanatory text, it is just a case of needing to have better documentation and strings that we can use for the filter operators etc. The catch is of course how do we include that without adding significant bulk to the file size, and we need to make sure it can be translated as well. Less of a concern if we think of this as a plugin, but it would add a lot of value as a core component.

We already have an "internals" plugin that might be the natural home for this?

Naturally we can do a lot better for the UI. One concern is that displaying the filter syntax breakdown and explanations side by side does not work in the fixed-fluid story layout (or on mobile). So I am considering moving the explanations below the filter syntax breakdown, where clicking on a part of the syntax highlights the appropriate description and vice versa (Somewhat inspired by http://apps.workflower.fi/vocabs/css/en#selector).

Conceivably one might be able to interleave things; this is very rough:

[saqimtiaz]

@Jermolene I'll try to find time to pick back up on this in a week or so. I need to update the code to handle multiple filter operands and filter run suffixes. I had also avoided adding start/end ranges to the filter parser and doing so now will simplify things.

The approach I've taken for now is rather crude but effective, iterate through the parse tree setting up some variables and transcluding different templates for operands, operators etc. The presentation is then just a matter of tweaking the wikitext templates. We can probably get something simple out the door first and see what user feedback is like, and then iterate on the UI to refine it.

We already have an "internals" plugin that might be the natural home for this?

This is was my first instinct as well. However the "internals" plugin seems to scare off the average user as being something development related. I would like for this to be more accessible (and to be available on tiddlywiki.com for people to play around with filters). A new core plugin introduced for this purpose could be an option, with an intuitive name that makes it easy to understand what the plugin offers with regards to filters.

[saqimtiaz]

Interleaving the explanatory text with the syntax breakdown:

[pmario]

I do like the view from https://github.com/Jermolene/TiddlyWiki5/issues/5058#issuecomment-836875197 much more, because it gives a much better overview.

The second approach has a lot of visual clutter that makes it hard to read

[pmario]

What if the text would be shown "on hover" in the same line as the different elements. ... I know that hover isn't something that can be used with mobiles. ... But it would be very effective for PC

On mobile, there could be a little button, which does the same thing.

[pmario]

So the mechanism should have a "line template" with which we can experiment.

[linonetwo]

With this tree, I think we can use https://github.com/google/blockly to make a visual builder/ visualizer.

[CodaCodr]

@linonetwo I could be wrong but... I predict blockly will go the same way as most visual programming tools: either die or occupy a tiny corner of the space they're meant to fill, rarely to be heard from again. They all fail for the same reason IMV. In "hiding complexity" they lose expressiveness. As children, we are not taught to "read" lego blocks or jigsaws, we are taught to read and express ourselves through/with text.

Just my opinionated opinion ;)

[CodaCodr]

@Jermolene @saqimtiaz

You might consider starting from the "end" and working back to the "beginning".

Reasoning: beginners are focused on the end-part of their wish. That part captures their ultimate wish.

  -- e.g., append "blah" to a set of tiddlers

Deriving the set of tiddlers via tags shared in common is not necessarily their first thought, appending "blah" is.

Of course, filters with N runs might prove a little more tricky, but they were going to be tricky either way.

[pmario]

Since our filters get more and more complex and the filter syntax is more and more powerful, I think we should give this issue a bit more love again. ... There is also some discussion going on about making the filter syntax more accessible, which imo will help all users, which this issue shows.

[linonetwo]

I'll experiment with dot-line editor like https://github.com/node-red/node-red in my job next month. Maybe I will have some inspiration about a visual dot-line editor about filter expression...

Visual builder can be friendly to new users, and can also be a syntax/semantic checker for old users.

[AnthonyMuscio]

Perhaps another way to address this issue is building easy to customise filters that are brought in using the filter or subfilter operators. Then the designer can just select from a set of available filters in an accessible way, or copy a macro and modify the field and tag names used.

Perhaps even a interactive way create and modify canned filters, to modify the tag, fieldnames or values. Elsewhere in the wiki you just use it as a subfilter. eg: "[subfilter<active-todos>]". I already have an example set of filters one may wish to use globally like this. Imagine if a configuration tool allowed to you select and modify an architype filter.

\define active-filter() [!tag[done]!tag[archive]]
\define inactive-filter()  [tag[done]] [tag[archive]] +[limit[1]]
\define active-todo() [tag[todo]filter<active-filter>]
\define active-project() [tag[Project]filter<active-filter>]
\define inactive-todo() [tag[todo]filter<inactive-filter>]
\define inactive-project() [tag[Project]filter<inactive-filter>]
\define created-today() [days:created[0]]
\define modified-today() [days:modified[0]]
\define due-today() [days:due[0]]
\define due-before() [days:due[-10000]]
\define no-due-date() [!has[due]]

Once there is a set of architype filters, selecting and using them in the wiki can be simplified and made more accessible?

[pmario]

Perhaps another way to address this issue

I do like this idea and I think it would already be of much help for new users, if the "subfilters" you mention would be explained in the docs. .... somewhere. ... Especially for filters there will never be enough examples. So every bit may help

[Jermolene]

Perhaps another way to address this issue is building easy to customise filters that are brought in using the filter or subfilter operators. Then the designer can just select from a set of available filters in an accessible way, or copy a macro and modify the field and tag names used.

Thanks @AnthonyMuscio that's a good idea, but I think it's a separate feature from the visualisation tool under discussion here. Also it's worth noting that the user defined parameterised filter operators in #6666 will give us much better tools for decomposing complex filters.

[AnthonyMuscio]

a separate feature from the visualisation tool

I agree, though part of my suggestion here is that the "architype filters" concept may be quicker and easier to implement. baring more fruit sooner, for novices using filters than the "visualisation tool". Regardless I think we can do both and again this will be easier after #6666 I will start a discussion. The “architype filters” collection

[linonetwo]

Visualize is not enough, I'm going to create editor using https://github.com/nocode-js/sequential-workflow-designer or https://github.com/Blackprint/Blackprint

But not until #8154 , so I think this will happened in half a year.

[CrossEye]

I created a followup to this discussion in talk.

It's simply another version of the sort of presentation Saq and Jeremy did above. It doesn't consider how to generate that presentation. But I think it's useful. You can download FilterBreakdown.json and add it to tiddlywiki.com to see it in action.

There are two variants. One is list-like, similar to the earlier examples:

The other is in a nested tree, which makes the hierarchy of elements more clear:

As I mentioned in talk, there would be a lot of work to get this working correctly, and it might depend on having source line/column numbers in the generated syntax tree. (Are they there already?) And it would depend on having a good mechanism for creating short descriptive strings included for all the built-in operators: no small chore.

But it might also be a useful tool for documenting how filters work.