Add support for comments within filters

[Jermolene]

In order to help make complex filters more readable, I'd like to explore the idea of supporting comments within filters via several alternate syntaxes:

<<mymacro """
[tag[One]] // This is a single line comment
/* this is a potentially multiline comment */ [tag[Two]]
<!-- and so is this another comment --> [tag[Three]]
""">

The idea would be that comments would act as a filter run, but not have any observable effects. Thus the following would not be recognised as containing a comment:

<<mymacro """
[tag[/* not a comment */HelloThere]]
""">

It's worth noting that the custom function support in #6666 can also help with making filters more readable by giving subfilters meaningful names and parameters. So perhaps a different approach would be to decompose complex filters into a chain of custom functions, each of which could be self documenting if the names are chosen carefully.

[kookma]

Good idea @Jermolene It happens many times to me where a complex filter needs to be commented and I had to add comments before or after it which one may do not find it easily!

I like something like below:

<$list filter="""[all[tiddlers+shadows]tag[$:/tags/Macro]] <!-- select all global macros -->
                 :map[{!!caption}!is[blank]else{!!title}]  <!-- return caption where possible -->
.
.
.

• I think one consistent form would suffice this. I propose <!-- to be consistent with current commenting syntax
• Simple is better, so for me, the below is confusing

<<mymacro """
[tag[/* not a comment */HelloThere]]
""">

[Jermolene]

Simple is better, so for me, the below is confusing

It is a contrived example. I'm trying to show that the /* comment */ syntax is not recognised within an operand. And so the above is literally saying give me everything tagged "/* not a comment */HelloThere".

[rmunn]

If there's a chance that the comment syntax could overlap with tiddler titles / tags / whatever that people have already made, then one possible solution could be to require opting in to filter comment syntax via \pragma filtercomments (or \pragma filtercomments on) at the top of the file.

But by requiring comments to only start in the filter-run position of the syntax, I think you avoid that issue entirely. The only drawback I can see is that sometimes I might want to insert comments in between steps of a single complex run... and since runs cannot have whitespace inside them, it would be hard to have meaningful comments in there anyway. So if a run is so complex that I want comments inside it, I should prefix it with a multi-line comment (or several single-line ones) explaining the run in detail.

[Jermolene]

If there's a chance that the comment syntax could overlap with tiddler titles / tags / whatever that people have already made, then one possible solution could be to require opting in to filter comment syntax via \pragma filtercomments (or \pragma filtercomments on) at the top of the file.

But by requiring comments to only start in the filter-run position of the syntax, I think you avoid that issue entirely. The only drawback I can see is that sometimes I might want to insert comments in between steps of a single complex run... and since runs cannot have whitespace inside them, it would be hard to have meaningful comments in there anyway. So if a run is so complex that I want comments inside it, I should prefix it with a multi-line comment (or several single-line ones) explaining the run in detail.

That's what I'm thinking too; it's very hard to imagine an existing filter that uses the exact syntax we require. There is perhaps a bit of a risk with dynamically constructed filters, but I think only if a tiddler title happened to precisely match part of the comment syntax.

[pmario]

I do like @kookma 's version. Let's keep it simple.

<$list filter="""[all[tiddlers+shadows]tag[$:/tags/Macro]] <!-- select all global macros -->
                 :map[{!!caption}!is[blank]else{!!title}]  <!-- return caption where possible -->

We would need to run tests with some "crazy" tiddler names, that look like filters themself.

[Jermolene]

I do like @kookma 's version. Let's keep it simple.

Just to be clear, you mean that we should restrict it to just HTML style comments?

I'm not sure I agree. There's a lot less typing in // comments, and a little less typing in /* comments. Having multiple comment formats makes it easy to allow people to comment out sections of filters that themselves contain comments.

[Jermolene]

Slightly off-topic, but immediately after posting the above comment, I saw this tweet:

Today I learned that HTML comments are valid in JavaScript! 😱 But only as a single-line comment. console.log('hello') <!-- Print hello --> console.log('world') ☝️This is perfectly valid JS but it only prints out 'hello' https://twitter.com/koddsson/status/1552958566700572672

[kookma]

Just to be clear, you mean that we should restrict it to just HTML style comments?

Yes, having several forms of comment/comment block increases confusion! Right now Tiddlywiki allows us to use comments inside wikitext macros, scripts, ... using the html syntax!

By the way! I also try to have best practice rules for myself. So when Tiddlywiki supports several forms of doing the same thing, I ask and look for best practice.

[rmunn]

There's a lot less typing in // comments, and a little less typing in /* comments. Having multiple comment formats makes it easy to allow people to comment out sections of filters that themselves contain comments.

Just to chime in, I'd vastly prefer // comments over anything else. My reasoning is that filters feel like a programming language to me, and nearly every programming language has converged on either // or # for its comments, with // being far more common than #. It's also way, way easier to type than <!--, which involves reaching for three different keys, two of them with the shift key pressed. Whereas // is just a single key tapped twice. (On some keyboards it might involve a shift or an AltGr which would make it more difficult than on the US keyboard, but even on those keyboards it's just the same key combination pressed twice).

[rmunn]

P.S. Another VERY good argument in favor of // is that it doesn't complicate parsing. With any comment that has start and end markers, you run the risk of mis-parsing an end marker and coming up with nonsense. E.g., here's a contrived (but not TOO contrived) filter:

[tag[todo]has[important]addprefix[LOOK HERE --> ]]

Note how the "arrow" used in the "LOOK HERE -->" text has used the same characters as the HTML closing-comment tag.

Now imagine this is part of a longer filter expression, which you're trying to debug because you got something wrong. So you want to comment out this sectiion of the filter run, to see how the output changes. How do you comment that out?

The "obvious" way to do it, which anyone with no programming experience would probably try, is to wrap that line with <!-- and -->, like so:

<!-- [tag[todo]has[important]addprefix[LOOK HERE --> ]] -->

As an exercise for the reader, try to write a parser that will find the second --> as the closing comment tag, and not the first. I think you'll quickly find that it can't really be done. Just about any parser, short of full-fledged finite state machine implementations, will match the first --> as the closing comment, leaving a filter expression that looks like ]] --> which is, of course, going to produce undesired results (probably an error). And who wants to write a full-fledged finite state machine for parsing comments? Not me, that's for sure.

But with // comments, the parser logic is easy. Any // that appears in filter run position means that all text from that point until the next newline character gets swallowed. No need to parse the inside of that text looking for closing comments. And, crucially, if you're commented out one section of code, you can comment out a larger section of code just by adding // to the start of the line again. So the inner section ends up looking like // // this was commented out twice, which the parser (and other tools) will have absolutely no problem with.

HTML-style comments might look like they're going to be simpler to understand. But they have some failure modes that turn out to be quite nasty in practice.

[Jermolene]

Thanks @rmunn

Just to chime in, I'd vastly prefer // comments over anything else. My reasoning is that filters feel like a programming language to me, and nearly every programming language has converged on either // or # for its comments, with // being far more common than #. It's also way, way easier to type than <!--, which involves reaching for three different keys, two of them with the shift key pressed. Whereas // is just a single key tapped twice. (On some keyboards it might involve a shift or an AltGr which would make it more difficult than on the US keyboard, but even on those keyboards it's just the same key combination pressed twice).

Yes indeed, there are compelling advantages to both the // syntax and the other two. As you note, the ability to nest comments gives one the potential to be able to comment out elements of a filter that themselves include comments is very useful.

It would perhaps be interesting to put together some test cases.

[kookma]

With respect to release of TiddlyWiki 5.3.0 in next future. I think this Idea: comments within filter worths to be added to the core. I am ready to help for testing and proving examples for documentation!

[pmario]

My concern is performance. ... Can we do a .compileFilter() and cache compiled filter-functions in a way, that comments do not have any effect.

... and ...

The current low-level, filter-call-stack makes single step debugging a nightmare already. IMO it will be even worse with a lot of comments

[bluepenguindeveloper]

I came up with another idea which I posted on Talk: a :noop filter run prefex with a shortcut such as #. This, combined with the existing quoted filter run syntax, would allow you to write comments like this:

#" This is a comment, implemented as a :noop filter run. "

Since this is just a new filter run prefix with a shortcut utilizing existing filter run syntax, I assume it wouldn't require any changes to how filter expressions are parsed, and thus would be much easier to implement and less risky/bug-prone than adding some kind of new syntax for the parser.

To undermine my own idea, perhaps there isn't any need; I currently just use the - prefix on quoted filter runs to write comments:

-"This works as long as none of the inputs are exactly the same as this comment; any such inputs would be removed."

Edit: when I say "my own idea" I do not mean it's "all my own"; in fact, the idea was in response to (and inspired by) @CodaCodr 's suggestion of having a noop[] operator (in the same Talk topic).

[pmario]

I came up with another idea which I posted on Talk: a :noop filter run prefex with a shortcut such as #. This, combined with the existing quoted filter run syntax, would allow you to write comments like this:

" This is a comment, implemented as a :noop filter run. "

I think that's a viable "comment shortcut"

There would be 2 or 3 versions

• #"comment comes here"
• #[[comment comes here]]
• :comment[[comment comes here]]

This should be straight forward to implement, since there would be no need for a new parser. Just 2 new operator prefixes will be needed.

[bluepenguindeveloper]

Single quotes can also be used as a filter run delimiter, so you could have #'comment comes here' as well, and they could be nested (though not indefinitely).

Edit: sorry for the double post - I deleted to recreate the comment as a reply, not realizing that the only reply functionality is the quote reply.

[pmario]

I did a short POC

• # or // and :comment .. are possible filter run prefixes.

• I'm not sure yet, but I think I do like // the most

[pmario]

preview edition is in the making ;) https://github.com/Jermolene/TiddlyWiki5/pull/8190

[Jermolene]

As per my comment on talk, which is consistent with my proposal in the OP:

I like the idea of allowing comments within filters, but I am not sure that this is the right approach.

The advantage of this proposal is that it doesn't introduce any new syntax to filters, reusing the existing syntax. The fairly devastating disadvantage is that comments need terminating, which doesn't match the user experience of most other languages.

Given that nowadays filters are often written across multiple lines, I think it would be worth exploring comment syntax that is terminated by the end of the line.

The most obvious approach would be to use // as the comment start symbol, ignoring all characters up to and including the following line break. A necessary subtlety would be that the character sequence would only be recognised in between filter runs.

It might be used like this:

\function edit-preview-state()
[{$:/config/ShowEditPreview/PerTiddler}!match[yes]then[$:/state/showeditpreview]] // Use per-tiddler state tiddler if present
:else[<qualified-preview-state>] // Otherwise use the qualified state
+[get[text]] // Get the text of the state tiddler
:else[[no]] // Default to "no" if not present
\end

The syntax could still be used with single line filters, but would require the comment to be at the end of the filter:

<$list filter="[[HelloThere]] // This is a comment" />

This syntax is not strictly backwards compatible. In current versions of TiddlyWiki, the example above evaluates to "HelloThere", "//", "This", "is", "a", "comment".

Note that with this proposal there would be no comment recognised in the filter [[HelloThere]] [[//]], so it would not break our ability to include "//" in filters.

[pmario]

New PR on the way. Preview should be finished soon:

• https://github.com/Jermolene/TiddlyWiki5/pull/8191

[Jermolene]

Thanks @pmario