org-ql link query does not find links whose description contains brackets

[exot]

Hey,

first of all: many thanks for org-ql! I am enjoying the way it let's me make queries on my org files and dynamically extract reports on complex tasks I have to take care of. That really takes my workflow to a new level :)

One of my main use cases of org-ql is to find items linking back to the current item (thus implementing some kind of "backlink listings"). I am mostly using the link query for this, but recently I discovered that this query did not find items I had linked back to the item at point. Going through the code I found that org-ql--link-regexp contains the following definition (among others):

      ((no-desc
        (match) (rx-to-string `(seq (or bol (1+ blank))
                                    "[[" (0+ (not (any "]"))) (regexp ,match) (0+ (not (any "]")))
                                    "]]")))
       (match-both
        (description target)
        (rx-to-string `(seq (or bol (1+ blank))
                            "[[" (0+ (not (any "]"))) (regexp ,target) (0+ (not (any "]")))
                            "][" (0+ (not (any "]"))) (regexp ,description) (0+ (not (any "]")))
                            "]]")))
       ;; Note that these actually allow empty descriptions
       ;; or targets, depending on what they are matching.
       (match-desc
        (match) (rx-to-string `(seq (or bol (1+ blank))
                                    "[[" (0+ (not (any "]")))
                                    "][" (0+ (not (any "]"))) (regexp ,match) (0+ (not (any "]")))
                                    "]]")))
       (match-target
        (match) (rx-to-string `(seq (or bol (1+ blank))
                                    "[[" (0+ (not (any "]"))) (regexp ,match) (0+ (not (any "]")))
                                    "][" (0+ (not (any "]")))
                                    "]]"))))

Those regular expressions all exclude brackets in the description part (when they consider the description part at all). I think this may lead to certain links not being considered when their descriptions contain either inactive timestamps or counters (i.e. "[2/3]"), both of which I am making heavy use of. I have tried to come up with some fix for this, but couldn't find a real solution (I repeatedly ran into errors like org-ql--select: Stack overflow in regexp matcher when replacing (0+ (not (any "]"))) with things like (regexp ".*")). Currently, I am have locally patched this function to ignore the description part altogether, as I am only searching for "id:…" targets.

Is my understanding correct that the regular expressions should allow brackets in the description part, or am I missing something?

Thanks again!

  Daniel

[alphapapa]

Hi Daniel,

Thanks for the kind words. I'm glad to hear that org-ql is useful to you.

Thanks also for reporting this problem. I thought that Org did not allow brackets in link descriptions, and AFAIK it didn't in the past (they were converted to { }). But I just tested on Org 9.5.2, and rather than being converted to braces, they are escaped in the URI part, and left as-is in the description part, like:

[[*Heading \[argh\]][Heading [argh]​]]

It looks like the setting of org-link-bracket-re in org-link-make-regexps shows how to do this properly now. (I would just use those variables, but Org has changed such variables out from under me enough times that I try not to depend on them anymore.)

So I think it should be a simple fix. I'll try to get to it soon.

[alphapapa]

Hi Daniel,

Thanks for your patience. This should finally be fixed now. Please let me know if it works for you.

Also, reading this again:

I am enjoying the way it let's me make queries on my org files and dynamically extract reports on complex tasks I have to take care of. That really takes my workflow to a new level :)

If you have time, would you mind sharing a few examples of how you're using org-ql like that? It could help me understand how the package is used by serious users, and it might be helpful as an example for other users.

[exot]

Hey,

thank you for taking care of this issue! If will try to check your changes as soon as possible and will come back to you :-)

As to some examples of how I am using org-ql: I am happy to share what I do with org-ql, but it might be a bit complicated to explain because for me org-ql is usually part of some larger workflow. I'll try to summarize my main applications:

• Dynamically generating lists of items in checklists

  (This is some vague kind of generalization of Org's stuck task feature, but more fine-grained to different types of Org entries.)

  For recurring tasks, I usually have some template checklist that I copy over whenever a new iteration is due (I manage recurring tasks as described here: https://karl-voit.at/2017/01/15/org-clone-subtree-with-time-shift/ ). This checklist may contain subtasks that ask to check certain kinds of entries in my Org files for whether they need updates or whether they are still relevant. I am using org-ql's dynamic blocks to generate those lists whenever possible.

  Indeed, one example is to check whether all recurring tasks (I tag them with :PERIODIC:) still have enough entries, where “enough” here means there is at least one sub-entry at least 35 days in the future:

  ,#+begin: org-ql :query (and (ltags "PERIODIC") (not (tags "HOLD")) (not (scheduled :from 1)) (not (children (or (scheduled :from 35) (ts-active :from (ts-adjust 'day +35 (ts-now)))))))
  | Heading | Todo | P |
  |---------+------+---|
  ,#+end:

  Then, in my monthly review, I use org-ql to generate this list and go through every PERIODIC task and add new iterations via org-clone-subtree-with-time-shift when necessary (so far, I haven't found the need to automate this, as this manual workflow is completely sufficient for my needs).

• Generate a table of all tasks planned in a given period

  The main use-case for this is to control my current workload.

  I have some custom-made function that leverages org-ql's ability to filter items based on attached dates (timestamp, scheduled, deadline). For a given time range, I collect all these items and display them in a dynamic block, mostly to check whether the work I am supposed to do still fits in the time I have available. If I have to much work to do, I complain to my boss and refuse additional tasks ;-)

• Collect backlinks and generate dynamic blocks for them

  (This is motivated by the Zettelkasten methodology.)

  I am a heavy user of links between Org entries, and I like to check backlinks to Org entries at point (or to some of its parent entries) to see whether there are related relevant tasks. I used to use a custom agenda view for this, but with org-ql I can directly generate the list of backlinks as a dynamic block in the item itself and edit this list afterwards. I even have a special function that inserts those backlinks together with the above mentioned checklist.

As you can see, I am a big fan of dynamic org-ql blocks as well as org-ql's simplicity when used in elisp code :-)

If it's helpful, I could try to explain some of the approaches in more detail, but it's hard for me to tell whether anything I wrote remotely makes any sense to anybody who's not accustomed to my workflow …

Best,

  Daniel

[alphapapa]

Hi Daniel,

That's a very impressive system you have. Thanks for sharing and describing it. I'd love to read a blog post describing it in more detail. Of course, there's never enough time to do everything, is there. :)

From looking at that, I think that you must be one of Org QL's most advanced users, so please do let me know about any ideas you may have in the future about how it could make your work easier.

Also, you might be interested in this: https://github.com/alphapapa/org-ql/issues/331

I also have some WIP branches for improving dynamic blocks. One of them is especially interesting, I think, as it allows a dynamic block to contain, rather than the usual table, an Org plain list of entries formatted using the same function that formats them for org-ql-view buffers, grouped with taxy as shown in #331. Since it's a plain list, the sublists can be folded, and they can be exported, etc.

[alphapapa]

I'm going to reopen this issue to remind me to follow up on some of the examples you shared.

[exot]

Hi Adam,

I finally found the time to test you fix and report back that everything works fine! (Indeed, testing was quick, but replying here took me far too long, sorry for that …)

That's a very impressive system you have. Thanks for sharing and describing it. I'd love to read a blog post describing it in more detail. Of course, there's never enough time to do everything, is there. :)

Thank you for your feedback! Indeed, I am currently trying – or rather struggling – to describe some of my workflows, mainly for myself to keep an overview over what I am actually doing 😉 But it turns out that what is easy to use is sometimes quite complicated to describe, at least for me, so I am afraid this will take even some more time …

From looking at that, I think that you must be one of Org QL's most advanced users, so please do let me know about any ideas you may have in the future about how it could make your work easier.

Thanks again! 😃 Scrolling through the documentation of Org QL, I still have the feeling though that I am not leveraging even a fraction of the full potential of Org QL, as there are so many things I haven't tried out yet. However, when I ever find that there is something I would like to do with Org QL but cannot find how to do it, I will come back to you 😁

Also, you might be interested in this: #331

I also have some WIP branches for improving dynamic blocks. One of them is especially interesting, I think, as it allows a dynamic block to contain, rather than the usual table, an Org plain list of entries formatted using the same function that formats them for org-ql-view buffers, grouped with taxy as shown in #331. Since it's a plain list, the sublists can be folded, and they can be exported, etc.

These are indeed fantastic pointers, I will check those out, thank you! While looking at #331, I also learned about your with-emacs.sh repository, which will also be a big help for me. Your repositories are a huge treasure chest!

Please keep up this great work!

  Daniel

[alphapapa]

Hi Daniel,

Thanks for the update, and for the kind words. I'm glad to hear that they're useful to you.