Search bar filtering improvements

9z0 commented 3 years ago

Negative filtering (NOT x)
Allow mixed filtering. For example, currently you can either search Text or +project but not Text +project.

9z0 commented 3 years ago

Also, OR operator (+project_x OR +project_y). The usual deal!

9z0 commented 3 years ago

Would it be feasible in the short term to add a NOT operator by Ctrl+click on a button?

For example: click: school (purple) ctrl+click: homework (another color or diagonal stripes to indicate active negative filtering)

means todos that are tagged school but not tagged homework.

I would appreciate it.

ransome1 commented 3 years ago

@9z0 it would not be a huge problem, but before I start with any further filter or search enhancement, it would need to have a proper conceptual basis.

At the moment neither of it has a priority for me as I am committing the little time I have to reduce technical depth and introduce proper test automation. If you need this feature urgently I can only advise you to fork sleek and try to add the minor enhancements by yourself. If they get implemented well and fit into the bigger picture, we can push them upstream later on.

9z0 commented 3 years ago

it would need to have a proper conceptual basis.

introduce proper test automation

👍

and try to add the minor enhancements by yourself

I would if I could! I can visualize the solution, but I don't have the programming skills, unfortunately.

I understand your priorities for the project and thanks for putting time into Sleek. It is not often that I take to an app like I did with Sleek.

ransome1 commented 3 years ago

That's good to hear. Even though not everything is being worked on at the moment, please don't get tired reporting new ideas and bugs.

Also the number of contributers is slowly growing and maybe at some point there might be others who will pick up the feature requests and start working on those as well. Let's see where this is heading.

zerodat commented 3 years ago

I have a possible solution to this feature request, as well as #108. I call it a "filter query language". I have a branch in my fork of sleek for this and I will make a pull request as soon as I finish describing this here. A full description with all the details deserves a wiki page, but here is a summary of what the code in the pull request will do:

If you type a string in the search box, it already performs a simple filtering process using an exact case-insensitive match of the string against the text of each todo. I am introducing a new syntax for an "advanced search". If the first character in the search box is a question mark, the search string will be parsed as a query language expression that gives you quite a lot of searching power. Some examples might explain it best:

?+manhattan or +forbin will filter for any todos with either project, "manhattan" or "forbin". ?not +forbin and @home will filter for todos that don't have the forbin project and do have the home context. ?priority > B or due < today-2w will filter for todos that have priority C, ..., Z or a due date before two weeks ago. ?not complete and due will filter for todos that aren't complete and have a due date. ?not +* and @* will filter for todos that have no project and have at least one context. ?priority and /h..e/i will filter for todos that have some priority and also match the regex h..e in a case-insensitive form, so for example, the text of the todo needs to include have or HOME or something like that.

Those examples give you some idea of the scope of this. If you can build sleek yourself, feel free to pull down https://github.com/zerodat/sleek.git and check out the filterlang branch and build it. Then you can try it out yourself.

That's what I've got so far. Looking forward to hearing what you think. Here are a couple of ideas I have for future work on this:

this filter query language can likely do any of the filtering that is being done in the filter panel. I would love to see a string field in the filter panel where you could specify an expression in the filter query language as a persistent filter, rather than a transient search string.
Even better, I'd love to have a scheme to specify a "view" as a saved filter query language expression, and let users define several of these views that can be quickly clicked on to change the filter (and perhaps we could add sorting to that as well).

ransome1 commented 3 years ago

@zerodat , that sounds fanastic and absolutely useful. As soon as I find some time, I will build your branch and try it out.

On top of the implementation we should think of how to educate users on this feature as it is more advanced as the rest. I'll give it some thought too.

9z0 commented 3 years ago

@zerodat Brilliant! Thanks a lot!

[x] case-insensitive
[x] And/Or
[x] Negative filtering
[x] Regex. (What is the minimum syntax? will /b/ be parsed as regex? How to avoid conflict with regular forward slashes in a todo?)
[ ] How would you handle grouping? A or B and C. Do you use parentheses? (A or B) and C vs A or (B and C)?
[ ] Is it freeform? For example, ? Will lore +home mipsum match loremipsum +home?

zerodat commented 3 years ago

@ransome1 I will be happy to write a whole page on this for the wiki documentation, and also as an embedded help within sleek's "?" page, where you have the info about shortcuts, recurrences, and so on.

Yes, @9z0 there is parenthesization, so your example:

A or B and C -- would by default be equivalent to A or (B and C) because "and" is the higher precedence operator. (A or B) and C -- would give you the other choice. The language supports full expressions of any length with parentheses and the operators:

! "not" "NOT" -- negation && "and" "AND" -- conjunction || "or" "OR" -- alternative

It is "freeform" as you asked, I think. So "+home" will match the project any way that is syntactically valid for todo.txt. As for your example, you would write it this way: "lore" and "mipsum" and +home, and that would match either lore +home mipsum or loremipsum +home. The order of the separate items in the filter expression doesn't matter, so you could get the same result with "mipsum" and +home and "lore".

The logical expressions that you can combine with these operators are:

+project @context complete priority due "case-insensitive literal string match" /javascript regex with optional i after it to make it case insensitive/

In the "string" and /regex/ you can escape the delimiter with backslash, so "quoted \" string". Single-quote strings work too. The escaped regex would be /[0-9]+\/[a-z]*/ for example.

In the due date case, you can say just due to test whether the todo has a due date. Or you can do a comparison of the due date against a date expression using ==, !=, <, >, <=, >=. The date expression can be today, tomorrow, a literal date string of the form 2021-06-22 or one of those with an interval added or subtracted: today+2b or tomorrow-11m. These are computed using the same algorithms I put in for the recurrences a while ago.

In the priority case, you can just say priority to test whether the todo has a priority. Or you can compare the priority with any of the letters A to Z using the ==, !=, <, >, <=, >= operators. This is lexical order, so A is less than B, even though it is "higher priority".

9z0 commented 3 years ago

@zerodat I couldn't be happier! Thanks again.

ransome1 commented 3 years ago

@9z0 is it ok with you if we highjack this issue and further discuss @zerodat feature in here?

@zerodat , I merged your PR, build the peggy scrips and started up sleek. Typing ? did only empy the table. So I guess the feature is not imported yet. Are there any further steps I need to do?

To avoid misunderstandings: My expection would be to write ? which would have no effect on the todo selection and then continue with something like "?+projectname" which would then narrow down all todos that actually hold projectname within their todo.txt objects. That is correct?

ransome1 commented 3 years ago

I need to take it back, it works :)

I think what got me confused are these first observations:

Typing ? immediately triggers the search and presents no results. That lead me to the believe, that it was not working.
Continue typing won't produce any results until in my case the input string exactly matched the context name: This could be a usability issue as the regular full text search narrows down the result with every keystroke. Maybe this could be adapted to this feature as well? Just with the difference that the search will be restricted to the attribute that has been prefixed with @, +, etc. Also I could imagine that some users might be starting typing in a project just with a specific letter in mind, that is part of the project's words, and would appreciate this auto complete like method.
The not method works a bit differently than regular method. Here I will type ?not+ which produces 0 results and as soon as I type in the first letter, all todos are shown, until the string matches one specific value

So I guess if we prevent the search until the first actual value is typed, this could solve this issue. For instance typing ?not+ could not trigger the search and with the first letter ?not+q the search fires and starts searching for projects that contain the letter q.

ransome1 commented 3 years ago

@zerodat , just a thought but do you think it might be possible to remove the ? out of the equation? Wouldn't it be great if you just type @ and sleek just knows that it has to search in contexts only?

zerodat commented 3 years ago

Hi, @ransome1. Thanks for trying this out! The general idea is that we need some way to tell sleek that we want to do an "advanced search" rather than the default search, assuming we want to keep the old behavior working. This could be a button or something, but I chose to use the ? as first character to make it easy to implement. Happy to see this changed to a fancier UI.

Once you choose an advanced search, it needs to be syntactically valid. Again, there could be a UI like turning the type-in area red when it is not a valid query. What I did for now is to make it NOT match if it is invalid. That way you get a clear indication that something is wrong.

Neither of these is necessarily where we would want to end up as a usability consideration, but good for trying out the language. If I have you guys convinced that this is a good feature to have, then I would be happy to work more on UI, or (even better) get your help with it, @ransome1.

Since the project and context are doing an exact match, you won't get matches with just a prefix of the project or context. I could easily change the semantics so that "+foo" will match projects beginning with foo. That's probably a good idea. The only downside would be that if you have two projects "+foo" and "+foobar" there would be no way to match only "+foo". We would need a way to say that we want an exact match, I guess. Maybe something like this: +"foo" if you want to force it to match only foo and not foobar? I'll try to implement these changes and see how it works.

If you use "not" then you get the complementary set of matches. So if +foo matches nothing then not +foo matches everything. In other words, it selects all todos that don't have a project "+foo". If we make the change to let the first part of the project or context match, then not +foo will match everything that doesn't contain a project that begins with foo.

zerodat commented 3 years ago

@ransome1 I pushed a new commit to my filterlang3 branch and made a new PR #175 . This should give you all of the things that we talked about. See how you like it!

Just to itemize the changes:

if you type just ? it will make a null query which matches everything.
if you type ?+ it will match everything with a project (just like +* does).
if you type ?+foo it will match projects foo and foobar.
if you type ?"+foo" it will match only the project foo.
as before, if you are in the middle of some syntax that is broken, it will match nothing, to show you that the query is not finished yet.

9z0 commented 3 years ago

@9z0 is it ok with you if we highjack this issue and further discuss @zerodat feature in here?

NOOO! It's my family heirloom! :P (Go ahead!)

@ransome1, @zerodat

Leave ? to avoid confusion by regular users (e.g. literal *). In fact, make it ?? to be on the safe side.
Add a setting "AdvancedSearchPrefix":"??". Then those who want it as default can remove it "AdvancedSearchPrefix":"".
Few notes/thoughts:
- As-you-type should be the default.
- Terminal * is always implied in an as-you-type system.
- Leading * is not always implied (harder on resources, many more results, and not as strongly intuitive). But with something like sleek, it should.
- Adding any asterisk to the word nullifies that implication:
  - foo (no plus) matches +foo, @foobar, foo, foobar and afoobar
  - foo* does NOT match afoobar. You need to make it *foo* (add an explicit leading *).
  - Similarly, *foo will match only +foo and foo (ending with foo).
- Typing with a negative filter:
  - !f will mean exclude any thing containing an f.
  - !fo will show a bit more (show back all excluded from above except those with *fo*.
  - !foo also means exclude any *foo*. And so on.
  - !foo* will leave afoobar shown; since you removed the implication of a leading *.
  - !*foo* will hide afoobar again.
- Operators +, @, etc should be outside the quotes: ?+"foo".
- Results should be empty only if actually no matches or syntax is broken.
- Optionally, we can break the implication with quotes:
  - "foo" will only match +foo and foo.
  - !"foo" will only exclude these two.
  - In case you want to use quotes for other reasons (maybe to escape something without using a backspace), you are going to have to add any asterisks yourself: !"*fo+o*" to mean exclude anything containing fo+o rather than just exact matches.
- Examples from above:
  - not +quick will hide quickwin projects (implied asterisks).
  - not +quick* will also hide them.
  - not +"quick" will only hide +quick.
  - As you type @improve, less and less results will show until only @improve todos are shown (inclusion getting more specific).
  - As you type not+quickwin more and more results will show (exclusion getting more specific).
  - *quick* is always redundant (just use quick) unless you use quotes.

zerodat commented 3 years ago

@9z0 thanks for the suggestions. Here are my comments:

Leave ? to avoid confusion by regular users (e.g. literal *). In fact, make it ?? to be on the safe side. Add a setting "AdvancedSearchPrefix":"??". Then those who want it as default can remove it "AdvancedSearchPrefix":"".

I agree that the AdvancedSearchPrefix could be useful. But I would leave the default as "?". You could set it to "??" if you prefer.

As-you-type should be the default. Terminal * is always implied in an as-you-type system.

With my latest changes, you get "as-you-type" iwith implied terminal behavior for the context and project matches. And in fact, as you type is what is happening in general in these "advanced search" queries. But given an advanced search language, you can't know what is coming so it doesn't really make sense in many cases. So unfortunately it is not as simple as just making a rule about terminal .

Adding any asterisk to the word nullifies that implication: foo (no plus) matches +foo, @foobar, foo, foobar and afoobar foo does NOT match afoobar. You need to make it foo (add an explicit leading ). Similarly, *foo will match only +foo and foo (ending with foo).

I'm not sure I get what you mean here, and I already have syntax to handle this. To do full-text match, you use quoted strings or regex quoted by slashes. One improvement would be to let a quoted string or regex at the end of the query be treated as implicitly closed even if you haven't typed the closing delimiter. That would give a better "as-you-type" behavior. I don't like the idea of using a to indicate that we are NOT doing a free-form match. That seems like the opposite of what typically means, for instance, in a regex or glob.

!f will mean exclude any thing containing an f. !fo will show a bit more (show back all excluded from above except those with fo.

That's what happens now, but you need the syntax !"f" and I'm planning to try to drop the need for the closing quote at the end of the line, so you could type: !"fo !"foo, etc and get as-you-type changes to that full-text search match.

Operators +, @, etc should be outside the quotes: ?+"foo".

Well, it is actually either one right now. +"foo" matches the project foo, but not the project foobar. "+foo" matches those four characters in a full-text match, so that will match +foo something or +foobar or whatever.

Results should be empty only if actually no matches or syntax is broken.

Yes, that's the way it works.

Examples from above: not +quick will hide quickwin projects (implied asterisks).

Yes, that's how it works now.

not +quick* will also hide them.

I don't think we need this.

not +"quick" will only hide +quick.

Yes, that's how it works now.

As you type @improve, less and less results will show until only @improve todos are shown (inclusion getting more specific).

Yes, that's how it works now.

As you type not+quickwin more and more results will show (exclusion getting more specific).

Yes, that's how it works now.

quick is always redundant (just use quick) unless you use quotes.

I still don't like the syntax that you are suggesting, and I suspect we don't need it, since we have quotes and an implied at the end if there are no quotes.

Again, thanks for suggestions and for thinking this through so carefully. I'm going to add the "no delimiter needed at the end" right now for full-text and regex patterns. Hope you don't mind me resisting some of these ideas, but feel free to keep trying to convince me. It would be great if you can get a version of sleek with this feature to try out. I think you might find that it works pretty well for you in most cases.

zerodat commented 3 years ago

OK, I've made a couple of changes and added them to PR #175 .

You can type a full-text or regex match without the closing delimiter and it will be valid. You still need to add the delimiter if you want to put some other expression after the string or regex. So:

?"here is some text will search for here is some text in the text of the todo. To also check that there is a due date, for example, you would need to close the string first: ?"here is some text" and due

You can also now add a trailing double quote to a project or context name to get the same effect as having the name surrounded in quotes. So:

?+foo matches any project beginning with foo.

?+"foo" matches only the exact project foo.

?+foo" also matches only the exact project foo. This allows you to decide to make it an exact match as you type, without going back and inserting the first quote right after the +. Should be more convenient if you are typing into the search bar.

Thanks for your feedback, @ransome1 and @9z0. These changes resulting from your questions and suggestions will help make it more convenient for the "as you type" case by accepting more incomplete queries as valid.

9z0 commented 3 years ago

Let's create a table: What will x match out of y:

Row	Query	+foo	@foobar	foo	foobar	afoobar	+afoobar
Plain
a	`foo`	1	2	3	4	5	6
b	`foo*`	1	2	3	4	5	6
c	`*foo`	1	2	3	4	5	6
d	`foo`	1	2	3	4	5	6
e	`"foo"`	1	2	3	4	5	6
Project
+a	`+foo`	1	2	3	4	5	6
+b	`+foo*`	1	2	3	4	5	6
+c	`+*foo`	1	2	3	4	5	6
+d	`+foo`	1	2	3	4	5	6
+e	`+"foo"`	1	2	3	4	5	6
Negative
!a	`!foo`	1	2	3	4	5	6
!b	`!foo*`	1	2	3	4	5	6
!c	`!*foo`	1	2	3	4	5	6
!d	`!foo`	1	2	3	4	5	6
!e	`!foo`	1	2	3	4	5	6
Negative Project
!+a	`!+foo`	1	2	3	4	5	6
!+b	`!+foo*`	1	2	3	4	5	6
!+c	`!+*foo`	1	2	3	4	5	6
!+d	`!+foo`	1	2	3	4	5	6
!+e	`!+foo`	1	2	3	4	5	6

9z0 commented 3 years ago

A few notes:

I don't mind the discussion. Thanks for being receptive!
Lets take regex out of the discussion for now for simplicity.
I was mentioning what I think are good ideas, whether they are implemented here or not. I didn't mean it as "these don't exist yet in your solution!".
The notes/thoughts above (except the quotes one) are used by other applications like the well-known Everything, including the asterisk behavior.
But given an advanced search language, you can't know what is coming so it doesn't really make sense in many cases. So unfortunately it is not as simple as just making a rule about terminal *.

I am not sure I understand. Why not? Can you give an example?
I'm not sure I get what you mean here, and I already have syntax to handle this. To do full-text match, you use quoted strings or regex quoted by slashes.
- I think ?string should be a full-text search by default. If that is the case, then what do you mean?
One improvement would be to let a quoted string or regex at the end of the query be treated as implicitly closed even if you haven't typed the closing delimiter. That would give a better "as-you-type" behavior.
- Won't that cause problems if the user decides to use another quote further down (or up) the search. What is the specific need for this that can't be solved in a standard way?
I don't like the idea of using a to indicate that we are NOT doing a free-form match. That seems like the opposite of what typically means, for instance, in a regex or glob.
- It doesn't! We are doing a free-form search, just on the side specified by the user and not both. Since the user decides to specify where to use the glob foo*. The algorithm doesn't assume, "Well, they didn't type it at the beginning too, lets assume they meant to!" If the user wants to specify *foo* they won't type foo*! Again, I didn't come up with this idea!
It seems you are making a distinction between full-text vs project/context syntax. Could you elaborate? You can use examples from the table! 😉 For example, won't a5, 56, +a6 all match?

Row	Query	+foo (1)	@foobar (2)	foo (3)	foobar (4)	afoobar (5)	+afoobar (6)
a	`foo`	+	+	+	+	+	+
+a	`+foo`	+	-	-	-	-	+

Thanks for the other responses. I generally agree but I just don't want to make this longer than it is!

zerodat commented 3 years ago

I think that you are expecting this to be full-text search by default with some additional magic to match projects and contexts. But that's not what it is. Once you begin with ?, it is a search expression with keywords and symbols that have specific meanings. Full-text search is also available by enclosing the text in a quoted string.

So by default, ?not is a keyword meaning logical negation. ?priority specifies the item's priority value. ?due specifies the item's due date. These can't match as full-text by default or there is no search language. To match full text, you give a quoted string containing the text. ?due will match any todo with a due date, but will not match the word due or duet in the text of the todo, only when it is a due date tag. Because it is not matching the due: tag; it is matching the existence of a due tag value, that is, a due date. It is intended to be used most often like this: ?due > today-1w and due < today+2w.

Examples:

?foo doesn't match anything ... it is a syntax error. So in your table, nothing in row a will match.

?"foo" matches any todo with foo in the full text.

?+foo matches any todo which has a project that begins with "foo". But not a "foo" just anywhere in the text, even if there is a plus before it. Like this is a todo containing a+foodwill not match, but this is a todo +food will match. This is your row +a. In the table, only +foowould match and only if it was syntactically recognized as a project by todo.txt rules. +foobar would match (if it was a project), but you don't have that in your table. +afoobar would not match.

?+"foo" or ?+foo" will match a todo containing the project foo but not the project food.

Another example: what does ?foo and bar mean? WIthout the ? it would be a full-text search for the string foo and bar. With the ? it is actually a syntax error because we have to assume that and is a keyword and foo and bar are unrecognized words. But you can do ?"foo" and "bar" which will match any todo containing both of the strings foo and bar anywhere in the full-text of the todo. Or ?+foo and @bar which will match any todo that has a project beginning with foo and a context beginning with bar.

9z0 commented 3 years ago

I see. Well, if that is your preference, I can't argue with that. But if it is about whether there is a solution or not, there is. Perhaps more than one.

For example, the Everything solution (See here) is to match anything as text until followed by :.

So, due matches any text. due: matches the syntax word only.

The point of this is that you can mix things freely without having to think much about syntax.

In your last example, is and necessary? Are ?"foo" "bar" or ?+foo @bar incorrect?

zerodat commented 3 years ago

Thanks for pointing me to Everything, which I hadn't seen before. I guess it is a Windows file search program, is that right? It has an interesting query language.

I think that you are familiar with orgzly, so I would also point out Orgzly search expressions and the more elaborate unimplemented query language that has been proposed for orgzly. What I have implemented is a search/filter language with similar goals, but hopefully it is more powerful and a little less cryptic than orgzly's language.

Of course there are an infinite number of possible search language syntaxes. If there were a usable standard in existence, I would have used it. At first, I was thinking of trying to use the same syntax as a SQL WHERE clause, but it wasn't really suitable, so I tailored something specifically for todo.txt tasks. Since we already had a full-text/plain-text search box by default, I wasn't really trying to extend that search. I was really trying to define a powerful filtering tool that could be used to define any desired filter. I think that the language implemented does that pretty well, so rather than just changing syntax at this point, I would rather have a few folks try it out for a while first.

In your last example, is and necessary? Are ?"foo" "bar" or ?+foo @bar incorrect?

Yes, as it stands, you would need to have an AND or an OR between them. Of course, it is possible to assume an implicit AND or an implicit OR, but I haven't done that, and it isn't clear to me which one would be most useful. I think it is less confusing to just specify either AND or OR.

9z0 commented 3 years ago

Thanks.

Everything is perhaps the most popular file searcher. It has multiple clones stating explicitly that they are made to provide a Linux equivalent to Everything:

It is very fast (even for an indexer) and if you try its search style, you'd know why you might want it everywhere! 😃

I literally find it easier sometimes to use its search syntax than to spend 5 seconds to try and remember the name of the file I want!

It is also an example of the style of app that I was discussing with @ransome1. Out-of-the-box experience is very straightforward and simple to use for anyone, but customizability and options through the roof and never forces itself on you. It just gets out of the way.

Orgzly

I do, to an extent. I like your syntax. It is intuitive. I am just offering tweaks!

rather than just changing syntax at this point, I would rather have a few folks try it out for a while first.

I understand (and wish someone would give me a Windows build to play along! 😔) but my main fear is that changing syntax midway is always a jarring UX.

implicit AND or OR.

AND. Definitely AND!

bar commented 3 years ago

@zerodat I guess you didn't mean to at handle me

ransome1 commented 3 years ago

@bar lovely!

@zerodat @9z0 I blocked v1.0.8 for this feature and created a test release based on your latest PR, so we can actually see how it works: https://github.com/ransome1/sleek/releases/tag/v1.0.8

@zerodat , if you provide a rough overview on how it works, I can add it to the test release page. Maybe others could be interested in this feature but don't want to read through this quite long thread.

zerodat commented 3 years ago

@ransome1 thank you for creating the test release! I am writing a wiki page right now to describe the feature, and we can add a pointer to that page in the test release page.

As I write the documentation, I think it would be a good idea to add pri as a short nickname for priority to save on typing in the search bar.

zerodat commented 3 years ago

@ransome1 I pushed that change up to my filterlang5 branch on github, branching off the v1.0.8 tagged commit. Feel free to add that into your branch whenever convenient. It is just the one file filterlang.pegjs which needs to be updated.

zerodat commented 3 years ago

@ransome1 I notice that the X has disappeared from the search box in my 1.0.8 build. Do you know why that would happen? I don't know of anything that I changed to cause this, and it seems to happen with or without the ? for advanced filter expressions.

ransome1 commented 3 years ago

@zerodat no, don't worry, that's just a CSS glitch that's already on my list.

zerodat commented 3 years ago

@ransome1 I pushed another commit to filterlang5. Supports matching any short form of priority keyword beginning with pri but also prio, prior, ..., priority. That will keep the table looking more stable as you type the word priority if you want to use the longer form. Probably unnecessary but it seems like a slightly better user experience.

9z0 commented 3 years ago

@ransome1 Thanks a lot for the build.

@zerodat, @ransome1

Now that I got a chance to actually experience the feature:

First, thanks for it!
Typing and with multiple filters is tiring. I strongly suggest it becomes optional (implied).
Bug: I think you allow only alphanumeric characters in projects/contexts matching. It should be any non-whitespace /S after the operator character, to match: +project.1 +a/b-testing etc.
Bug (known?): Asterisk doesn't work: +foo*, +*ooba*.
Typing +1 should match +project.1.
It is easier to type due: to match any due date than to keep quoting regular in-text string.

For example,

(A) Pick up new graphics card due:2021-06-26 +components +trip
(A) Review graphics card due:2021-07-10 +components +home
(C) Sell old graphics card due:2021-07-30 +components +trades
(B) Write wiki for search feature +sleek 
(C) Write new graphics card experience blog post +components +blog due:2021-08-15
(B) Clean house +home

To search for:

Text: graphics
Project: +home
Has a due date

"grap" and +hom and due vs grap +hom due: or just grap me due:

You can also short due to d but that applies to both current d and suggested d: solutions.

zerodat commented 3 years ago

Hi, @9z0 . Here are some answers:

Yes, I should probably extend the alphabet for project and context names. I'll try to do that soon.
No, I don't have any special meaning for * any more, except for its use in a regex. And as in the previous point, * should be a valid character in a project or context name anyway.
The +xxx syntax matches a prefix of a project name to make it easier as you type, but I didn't intend to match substrings other than prefixes. So +1 won't match +project.1.
to match any due date, you can just type due because that's a keyword. You don't need to put it in quotes and you don't need a colon.
I see that you are trying to optimize for typing the minimum number of characters, but I am concerned with clarity as well as economy, so I hesitate to make the language more cryptic to make it more compact. Your grap me due: is half as long as "grap" and +hom and due but it seems twice as confusing. You might convince me to make the and implicit -- I'm still thinking about that. Then it would be "grap" +hom due which is not much longer. Not sure what you intend with me in the last version. Why not +h which is just as short and already works?

9z0 commented 3 years ago

Thanks for replying!

I appreciate your patience. I should clarify two things: what I want to achieve and what I mean:

Why support also matching from the middle (me for +home): 1. long projects, 2. similar beginnings. In the above example, +trip and +trade, or +project.1, +project.2 but in reality they are even longer and multiword. Note that people add prefixes when they name projects to group them together in the sidebar/suggestions/etc. Also see point 3.
I am mostly describing a shorthand, not the full-form syntax. It won't obviously be the only way and it won't confuse the person using it!
- grap me d: is just the unverbose equivalent of graphics +home due: or graphics and +home and due:*, which should obviously be the default supported. You use whatever you are comfortable with, the same way you use either ! or NOT.
- I could have used +me to specify searching in a project but I was fine with matching any me. I was just picking the most unique part of the word, which is the only me in the whole list, which is also why +h may not be the most unique, if I also have +house, +hording, etc

The +xxx syntax matches a prefix of a project name to make it easier as you type, but I didn't intend to match substrings other than prefixes. So +1 won't match +project.1.

I see the value here (+foo meaning +foo* and not +*foo*). You can't reconcile both the +foo* and +*foo* behaviors and if more users expect/use the first, then I can accept that as the default behavior. But I would at least ask for support for typing +*foo*.
- Also, wildcard support *, ? is expected by a lot of people and provided by many tools. So I think it should be enabled in advanced search.

zerodat commented 3 years ago

@9z0 you are wearing down my resistance. :smile: Here's what I have changed in response to your points:

the project and context names can now contain any non-blank symbol except for ". So you can have your +project.1 or your +a/b testing project. That is in keeping with the todo.txt syntax supported by jsTodoTxt, so I think it is the right thing. Thanks for pointing this out.
I've changed the default for project and context matching to match any sub-string of the name, so for instance +1 will match your project.1. If you put a double quote at the end, it will only match the exact name that you have typed, so for instance, +a/b" or +"a/b" will only match a project a/b but not a project a/b-testing.

I'll push these changes to github soon. But note that I haven't done everything you were hoping for. In particular ...

I am still not in favor of the wildcard support that you described. Of course, you can do all of that in the regex feature of the language, but not in project or context names.
You do still need to put literal text to match in quotes, like "grap".
I haven't decided whether to do the implicit AND operator yet. It would be a little strange, because AND has higher precedence than OR, so A or B C or D would end up meaning A or (B and C) or D which is probably not what was intended.

9z0 commented 3 years ago

@9z0 you are wearing down my resistance. 😄

I am not sure if that's a compliment or a complaint, but I'll take it! 😁

the project and context names can now contain any non-blank symbol except for ".

👍

I've changed the default for project and context matching to match any sub-string of the name, so for instance +1 will match your project.1. If you put a double quote at the end, it will only match the exact name that you have typed, so for instance, +a/b" or +"a/b" will only match a project a/b but not a project a/b-testing.

Great! Thank you.

But note that I haven't done everything you were hoping for.

That means I just have to try harder! 😛

I am still not in favor of the wildcard support that you described. Of course, you can do all of that in the regex feature of the language, but not in project or context names.

I think I can manage. As long as partial matches are on by default, wildcards drop in importance. If I need something like +*bar (match ending, which isn't possible in he current iteration), I can use regex. I'll report back if there are issues.

You do still need to put literal text to match in quotes, like "grap".

Mmmm... I will just say that perhaps most query langs (not just Everything) do plain text unquoted and syntax keywords with a separator:

Windows search: regular string keyword:value "literal string", with implied AND.
GitHub: string author:x assignee:y "literal string" with implied AND.
VS Code (as in settings search): @feature:xxx string with implied AND.
Web search: string filetype:pdf "literal string" with implied AND.
etc.

So, 🤷‍♂️.

I haven't decided whether to do the implicit AND operator yet. It would be a little strange, because AND has higher precedence than OR, so A or B C or D would end up meaning A or (B and C) or D which is probably not what was intended.

The frequency of AND in search is much higher than OR. When you search pizza recipes you want results that contain both pizza and recipe.

If you need both, you specify both, with the parentheses. I am not sure how to sort the ambiguous syntax A or B C or D but if I would automatically (without much thinking) use both AND and parentheses exactly because I want to avoid ambiguity. (A or B) and (C or D) or A or (B and C) or D People even do so with math just to be sure 3+(4*2), unless they know the notation the calculator uses. I don't think it is normal for someone to type it that way. The ambiguity produced by selective use of syntax operators is on them IMO.

ransome1 commented 3 years ago

I admit I did not read the whole thread in detail as it became too much of a deep dive into functionality I won't ever use.

Please keep in mind that all this complexity needs to be developed and this means days of work, testing and maintaining.

I don't want to talk you out of the deep dive, but there is a chance you're developing it basically for yourself. Users who want to use complex search search strings will most likely not be eager using an Electron app with a nice GUI to manage their todo.txt files. They will be using the todo.txt command line app.

If you want to create a feature for the whole sleek community, your time is spend more wisely if you think rather in every day use cases than in every possible way to exclude and include information.

A common use case: I am working on a project and I am looking for important todos within this project, that need to be done asap and not someday: +project AND @asap AND (A) AND NOT @someday

Believe me, it won’t need to get more complex than this in a todo app. Users want to get their things done, they don’t want to get lost into complex ways of searching for todos. This is not VSCode, Atom or any other tool which could require complex search strings. It's a simple todo app.

And by keeping it simple you don’t even need an additional trigger ? (which is not known in the basic context of todo.txt and something a user would need to learn on top only for sleek) to start the advanced search.

A seamless and user centered approach would let the advanced search start automatically whenever it needs to be started. For example by typing space and + or space and @ or space and ( and a capital letter or space and due:, and so on. The user would not even realize that there are two different search methods, it would feel like one that has been very well designed.

Also the auto complete function (although not perfect yet) has been set up to work like this and is already learned by users.

zerodat commented 3 years ago

@ransome1 Thanks for your comments. I don't think you need to read the thread in detail. This discussion about the language design has been quite long, but it has led to some improvements. I have already implemented those improvements, if you look at the filterlang5 branch in my fork of sleek. Unfortunately it seems like I can't figure out how to do a clean pull request relative to v1.0.8, so if you would just get the latest version of filterlang.pegjs and filterquery.mjs from the head of filterlang5, that would be great.

I think that the features of the current filter language are reasonable and useful. As you might have noticed, I have been writing documentation for it in the wiki.

Your example of the common use case +project AND @asap and (A) and NOT someday would be +project AND @asap and pri==A and NOT "someday" in my notation. Is that acceptable to you, or are you suggesting that I change the syntax some more?

On the subject of the ? trigger, I think that we do need some way to distinguish between advanced search and the default literal match of whatever string you type in the search box. We need to know whether the word and is the word and or the logical operator and, for instance. It doesn't have to be a ?, but we need some way to know.

I get that it would be nice to use only the advanced search, but still let users type literal words to match without quotes around them, But how do we know whether NOT someday means "look for the string NOT someday" or it means "look for todos without the string someday"?

ransome1 commented 3 years ago

@zerodat I recreated the release based on your filterlang5 branch, thanks a lot: https://github.com/ransome1/sleek/releases/tag/v1.0.8

Today I will propably not find any spare time looking into it. But I gave it a quick try and it works great :)

One thing I wanted to ask you though. When I type ?@context it will present all items with that context. If I further add a space the results will remain. If I go on and add and AND (?@context AND) , the results will be emptied. Only if I continue and add another prefix (?@context AND +), it will show the further narrowed down results.

If the AND operator will empty the results I assume users will think that there is no AND operator and they won't continue typing. Do you think it is feasible to skip the search trigger as long as the user is typing relevant operators like AND or OR?

zerodat commented 3 years ago

Thank you again, @ransome1. That's a good question. I can think of a couple of ways to approach this.

First of all, can we think of a way to avoid rebuilding the table with generateTable when the user types another character? I would guess that this would be very hard because we don't know what has been typed at that level.

Second, we might be able to handle this with memory in filterItems. Maybe we could remember the last query that worked and if the current query is a prefix of it, then we could present the same results as the working query. I'll try to implement that today and see how it works.

zerodat commented 3 years ago

OK @ransome1 I have some good news!

First, it worked fine to do what I suggested with remembering the last query and the result is a better user experience when typing a query in the case you described: ?@context AND.
Second, I tried a change where it defaults to literal string match if the query isn't valid (and if you aren't just extending a shorter query). This lets you type a plain text match or a search expression and tries to do the right thing by default. It isn't perfect: if you type a query ?duet it will not work because it will match for due date and then it will think you are still figuring out the next part of the query when you type the t. So it won't switch to literal matching in that case. Of course, you can skip the ? or use ?"duet" to match it exactly.

Seems pretty smooth to me. I'd like you to try it when you can. The change is at the head of filterlang5 again, and it is a change to filters.mjs only.

9z0 commented 3 years ago

@zerodat I really do appreciate your work and this is already better than what I as willing to settle for when I started this issue. So, with that in mind, I think that insisting to avoid a : or something like it to indicate keywords is making things a bit too confusing and "tailored". Going with due: would mean: no need for quotes (on one side" or "both"), no need for the algorithm to guess what the user is thinking at the moment, and intuitive/matching the universal way that the user is familiar with.

I understand that this is your decision and this is your contribution. You don't owe us anything. Just, keep us in mind. :)

+project @context string due: is closer to the way todo.txt itself works and makes things easier to understand and remember. A user intuitively knows to quote literal string if they need it "literal string". Most are familiar with A and (B or not C). etc.

I am not sure I can add much more without repeating myself.

Maybe other users can add their opinion?

zerodat commented 3 years ago

@9z0 I'm probably going to leave it as it for now. I think that adding colons or other symbols to the keywords doesn't solve the fundamental problem. The problem is that you still can't be sure of the user's intent, that is, did he mean to use an advanced search expression or just a literal string match. With my latest version, I switch to literal string match if the query doesn't seem to be an advanced search expresssion. In the rare cases where that doesn't work, the simple thing is to say "if you want literal text matching and it doesn't seem to be working for you, just begin your query with a double-quote." That will always work.

I'm on the verge of suggesting that we can now drop the ? at the beginning and just default to advanced search. In the worst case that would mean that the user must type a " to begin a literal search rather than a ? to begin an advanced search. But in many cases it will just do the right thing without needing either symbol. I hope you get to try it soon.

ransome1 commented 3 years ago

@zerodat , I'm just trying out your latest improvements and it works incredibly well. You did a fantastic job on this.

I updated the release, so everybody can check it out: https://github.com/ransome1/sleek/releases/tag/v1.0.8

But in many cases it will just do the right thing without needing either symbol.

That's the spirit. I want to post a heart emoticon, but I just can't figure out how to add it :)

I hope you get to try it soon.

I'm really excited to see if this actually works.

zerodat commented 3 years ago

@ransome1 I have pushed another commit to filterlang5 branch. Updates to filterlang.pegjs and filters.mjs.

This removes the use of ? to get the filter expressions. Instead, it always tries to parse as a filter expression, but drops back to literal string match if it can't parse. This is the change to filters.mjs.

The change to filterlang.pegjs fixes a possible bug that I introduced when extending the alphabet for project and context names. It was including parentheses in the project and context names which seems unnatural when you try something like (+bills or @home), It is natural to put the closing parenthesis right after the context name, but the parenthesis was being eaten up as part of the context name home), I don't think we want that. What do you think? todo.txt would allow parentheses in a project or context name, so this is debatable. Without this fix, you would have to type a space after the project or context pattern before the closing paren. I can remove this change if you guys think that it is better to include the paren in the name, but it seems unnatural and a bad UX to me.

9z0 commented 3 years ago

@9z0 I'm probably going to leave it as it for now. I think that adding colons or other symbols to the keywords doesn't solve the fundamental problem. The problem is that you still can't be sure of the user's intent, that is, did he mean to use an advanced search expression or just a literal string match.

How do you figure? Can you give an example (on how the proposed solution would still not solve it)? In fact I think it does the opposite. It removes ambiguity.

(+bills or @home)

Using syntax keywords/operators as literal characters within something should be quoted by the user. Priority should be given to close the parentheses. If they decide to have a parenthesis-containing project, then it should be quoted (+bills or @"home)")

9z0 commented 3 years ago

Here is a light-hearted challenge: I will give you syntax, and you tell me if you find a point of failure. 😉

Assumptions

advanced search starts with a ? like you have. Let's ignore adding this in the following points for simplicity. Assume it always exists.
For simplicity, I will ignore contexts. What applies to projects applies to contexts but with a @.

Syntax

Anything is a plain string unless otherwise specified by special syntax.
Projects: prefix with +, contexts with a @.
Wildcards: * means zero or more characters. ? means one character.
Any key:value syntax in todo.txt (due:<date>, rec:d, etc) is also searched in the same way.
Special key words:
1. priority will have the syntax priority:
  - values:
    - A, B, C, etc
    - with arithmetic syntax below (<=C)
    - ranges: A-D or A..D
2. completion status will have the syntax complete:
  - values: Boolean
    - yes, y or 1
    - no, n or 0
Double quotes (") are used to enclose literal strings. Any printable character is allowed within quotes.
- If a search string has a literal quote character, it is escaped by adding another one immediately preceding it "". For example: to search for the string lor"em, the syntax islor""em.
- All quotes must be matched. No left quote without a right one.
- Any even number of contiguous quote characters in search string is to be treated as a single literal quote character that has no syntactic function for each two given quote characters:
  - lor""em is interpreted as a literal lor"em.
  - "lor""em""""" is interpreted as: active left quote + literal string + literal quote + literal string + literal quote + literal quote + active right quote. This means two active quotes enclosing this literal string: lor"em"".
  - This weird example is to stress test the logic, You can go even weirder if you want. There should be no ambiguity.
  - In fact there is no need to surround this example with quotes since you can just escape the internal quotes by doubling them: lor""em"""" -> the same literal lor"em"".
Unquoted whitespace is to be treated as a separator between searchable units.
Parentheses are used to enclose any number of searchable units and to treat these units inside as a single search condition.
Logical and arithmetic operators are supported as appropriate:
- AND, OR, NOT (and their short forms &, |, !)
- ==, !=, <, >, <=, >=.
Any:
- syntactically relevant character (except the double quote): like (, ), :, , + etc.
- syntactically relevant word
that is quoted (alone or within a larger string) is considered literal. . ")", or "lorem (", etc.

Abbreviated syntax:

AND is implied. Searchable units separated with white space only are treated as both mandatory.
If no * is specified within an unquoted string, the searchable unit is assumed to have both preceding and trailing asterisks.
Short forms:
- due: == d:
- priority: == p:
- rec: == r:
- complete: == c:
- yes == y == 1
- no == n == 0
If no value is supplied to a keyword. The widest match is shown:
- due: ==due:any == due:* == due:?. Any todo with a due date.
- !due: == ... . Any todo without a due date set.
Unless a syntactically relevant character satisfies its positional conditions, it is to be interpreted literally. For example, lor+em does NOT need to be escaped, since the project plus sign needs non-word before it.
- Note that quotes have no positional requirements: +lor"*"*m means lor + a literal * (not a wild card) + an actual wildcard + m. So it should match the literal string: lor*sssem.

TLDR

This may seem complicated because of all the descriptions, but is actually very simple:

search a literal quote character by doubling it "".
search any other literal character by quoting it. Quote anything you doubt; it will work.

Valid Examples (All will work)

To match this todo: (A) Unit 1 "Thermodynamics" First 10 pages @homework due:2021-06-29 +physics

?@homework and due:* and thermo* is the same as ?@homework due: thermo is effectively the same as ?@homew d: ermo

You can also try: ?phys therm d: or (Unit, due date, homework) ?*nit 29 @*mew* or just ?nit 29 mew. or (literal quote character) ?unit "" phys

Unless I forgot a part, this should be it. Fire away!

ransome1 commented 3 years ago

@zerodat I updated the builds https://github.com/ransome1/sleek/releases/tag/v1.0.8

And I must say it really works well.

I ran into two minor things:

If you just type ! it will trigger the search with no results. Would it make sense to only trigger the search after the next input after the ! to avoid the empty result page?
If I type !due:2020 it will exclude all todos with due dates, even those with with due:2021

But again, these are just minor, the search itself is a lot of fun and I can't thank you enough and I'm looking forward to releasing it any time soon.

Right now I am refreshing the gui a little. I'll try to think of a way to indicate the main benefits of this somewhere in or close to the search input field itself. Let's see what I can come up with.

zerodat commented 3 years ago

Thank you, @ransome1. I think I know what is happening in your comment (2.). When you type !due it means exclude all todos with due dates, and then as you continue to type, the colon makes the query invalid, so it keeps the last valid query. It is just showing you the results of !due. You probably want to either type !"due:2021 or else !due>2021-01-01. Either should work. The first is the literal match against the text of the todo and the second is the date comparison for dates after the start of 2021.

I've been trying to think of a way to give the user some more feedback about the query processing. For instance, could we draw a green line under a valid filter query expression, and underline only the part of the query that is valid. In the query !due:2021 it would underline !due in green, so that you have an indication that it is ignoring the rest of the query because it is invalid. That might be an interesting thing to try. If I have time tomorrow, I might try to figure out how to do that.

As for comment (1): the result page is empty because no todos have an exclamation mark in them. If some did, you would be seeing those in the results. We don't know whether the user is really searching for exclamation marks, or has just typed the first part of a filter query expression, but since it isn't syntactically valid, sleek tries the literal text match of !.

To do what you are suggesting (avoid the empty result page) we could handle queries that start with ! ( or not) specially, and treat them as empty queries until we see what comes after them. The user would have to type "! to search for exclamation marks. I'll give this some more thought as well.

ransome1 commented 3 years ago

@zerodat I was hoping to be able to finish a GUI refresh and release it together with the advanced search. But due to my actual job and private life I won't be able to do this any time soon.

What we can do though is to release the advanced search without any educational feature within the GUI (that would basically mean bumping up the v1.0.8 pre release to a proper release) and just see if any feedback pops up.

My testing so far didn't came up with any interference of this feature with other functions and it should be safe to release. I think the worst case is that some advanced user inputs could lead to confusion which then hopefully turns into user feedback here on Github with which we can work.

Would you also say the feature is ready for deployment?

ransome1 / sleek