[Feature request] Link to heading without custom_id (headline search)

[jarofromel]

Hello,

I immediately liked the option to link to headings, thank you for that. I would be nice to have it without the custom_id boilerplate. Similar to the new option in the org-mode 9.7:

New option org-id-link-consider-parent-id to allow id: links to parent headlines

Storing a link with point at "Child 1" will produce a link <id:abc::*Child 1>, which precisely links to the "Child 1" headline even though it does not have its own ID.

Also the terse description in the manual:

id ‘id:B7423F4D-2E8A-471B-8810-C40F074717E9’ ‘id:B7423F4D-2E8A-471B-8810-C40F074717E9::*task’ (headline search)

The following paragraph is my opinionated user experience with custom_id, not critics in any way.

After several of weeks I stopped to use the links to headline with custom_id. It seems to me it goes against the Denote design simplicity which I like a lot (kind of polluting my notes with the boilerplate around custom_id which only goal is to link). I returned to org-mode way file:projects.org::*task title which is without the boilerplate (but it means also without Denote goodies).

[protesilaos]

From: jarofromel @.***> Date: Sun, 15 Sep 2024 11:03:12 -0700

Hello,

Hello there!

I immediately liked the option to link to headings, thank you for that.

You are welcome!

I would be nice to have it without the custom_id boilerplate. Similar to the new option in the org-mode 9.7:

New option org-id-link-consider-parent-id to allow id: links to parent headlines

Storing a link with point at "Child 1" will produce a link <id:abc::*Child 1>, which precisely links to the "Child 1" headline even though it does not have its own ID.

Also the terse description in the manual:

id ‘id:B7423F4D-2E8A-471B-8810-C40F074717E9’ ‘id:B7423F4D-2E8A-471B-8810-C40F074717E9::*task’ (headline search)

I see. I was not aware of this option. I wonder what is the workflow for it though, given that an ID (or CUSTOM_ID) link is, in principle, always unambiguous but the headline search can fail to go to the right place (I had plenty of cases like that). Plus, the headline search does not export nicely, e.g., to HTML.

The following paragraph is my opinionated user experience with custom_id, not critics in any way.

This is fine.

After several of weeks I stopped to use the links to headline with custom_id. It seems to me it goes against the Denote design simplicity which I like a lot (kind of polluting my notes with the boilerplate around custom_id which only goal is to link). I returned to org-mode way file:projects.org::*task title which is without the boilerplate (but it means also without Denote goodies).

Note that the CUSTOM_ID is a standard Org feature: we are not inventing anything Denote-specific here. What Denote does is simply streamline the creation of those CUSTOM_ID and then format 'denote:' links accordingly.

Context links are already supported, given that the 'denote:' link type is as close to 'file:' as we can make it. But we do not automatically generate those, because (i) they are more likely to break and (ii) do not export to reliable links, such as in HTML.

I am happy to review how this works though. But we need to have a clear idea of what we want to provide as user options and/or commands.

Also, we have to keep in mind how this will work with heading-specific backlinks.

-- Protesilaos Stavrou https://protesilaos.com

[jarofromel]

New option org-id-link-consider-parent-id to allow id: links to parent headlines

Storing a link with point at "Child 1" will produce a link <id:abc::*Child 1>, which precisely links to the "Child 1" headline even though it does not have its own ID.

Also the terse description in the manual:

id ‘id:B7423F4D-2E8A-471B-8810-C40F074717E9’ ‘id:B7423F4D-2E8A-471B-8810-C40F074717E9::*task’ (headline search)

I see. I was not aware of this option. I wonder what is the workflow for it though, given that an ID (or CUSTOM_ID) link is, in principle, always unambiguous but the headline search can fail to go to the right place (I had plenty of cases like that). Plus, the headline search does not export nicely, e.g., to HTML.

Looking at my denote notes I see that the headings are usually on the first level (* heading 1) and the content of headings is quite short (3-15 lines). I go very infrequently deeper than the level one. Thus the combination of the identifier and the heading title gives kind of URI in my case. The shortness of the headings means that adding CUSTOM_ID (3 lines) increases the content of the heading by 20%-100%. Typical examples is the denote file with the list of:

• books to read (heading=book title+author)
• short usage notes about emacs/python/etc packages (heading=package name)

Of course I have some files where headings go deeper. Like the discussion of the book, incremental understanding of some topic and here ambiguity could happen (* chapter 1, * lessons learned; chapter 2, ** lessons learned).

Linking to the first type of notes (short ones) happens far more frequently and the direction is usually from the second type (discussions) to the first. It is like the second type is build on or developed from the first one.

Also I find myself to browse thru my notes without org-mode machinery quite frequently (mobile, terminal) and Denote minimalism helps a lot. A hypothetical CUSTOM_ID in each heading of the first type notes introduce feeling of visual noise.

Context links are already supported, given that the 'denote:' link type is as close to 'file:' as we can make it. But we do not automatically generate those, because (i) they are more likely to break and (ii) do not export to reliable links, such as in HTML.

This is perfect. I will use. It solves a lot for me. Is it mentioned in the doc, please? I didn't find it.

I am happy to review how this works though. But we need to have a clear idea of what we want to provide as user options and/or commands.

In my case a simple option "link to heading by custom_id/context" is enough as it cover 95% of my link use. But I understand it is just narrow user experience. There are corners like "if there custom_id, use it, otherwise use context" which need to be answered. I will check the way the Org-mode handle these situations with ID.

[protesilaos]

I created a new branch that should provide the functionality you need. Can you give it a try? Link: https://github.com/protesilaos/denote/tree/org-context-links-for-headings

[protesilaos]

I also tested for per-heading backlinks. It seems to work fine in all cases. Maybe the code can be improved a bit, but it is okay for now.

NOTE: If there is a CUSTOM_ID it takes priority.

[jarofromel]

I tested the branch but the results are always the same. No matter what the value of denote-org-store-link-to-heading is I always get only the identifier (no context or CUSTOM_ID) in the the link via org-store-link and org-insert-link. And the denote-org-extras-link-to-heading always link to CUSTOM_ID.

But I am suspicious of my config, I use doom-emacs. When I returned to Denote main brainch the behaviour of org-store-link was also to return only the identifier in the link (no matter of the value of denote-org-store-link-to-heading).

I will check again tomorrow I'll be on another computer with vanilla Emacs.

[protesilaos]

From: jarofromel @.***> Date: Tue, 17 Sep 2024 07:41:01 -0700

[... 12 lines elided]

I will check again tomorrow I'll be on another computer with vanilla Emacs.

Fine. Just let me know how it goes. Everything seems fine on my end, though I might have missed some edge cases.

Also, please tell me if what is being created is an ID or CUSTOM_ID. The latter comes from Denote, but the former is not our doing.

-- Protesilaos Stavrou https://protesilaos.com

[jarofromel]

I've tested on the fresh install of emacs 29.4, mixed results I would say:

for (setq denote-org-store-link-to-heading 'context):

• denote-org-extras-link-heading
  • heading has CUSTOM_ID - OK - links to [denote:ID::CUSTOM_ID]
  • heading without CUSTOM_ID - NOK - goes thru selection process (select file then heading) but after confirmation do nothing
• org-store-link & insert
  • heading has CUSTOM_ID - NOK - link created but as [denote:ID::context]
  • heading without CUSTOM_ID - OK - as expected [denote:ID::context]

for (setq denote-org-store-link-to-heading 'id)

• denote-org-extras-link-heading - OK - links to [denote:ID::CUSTOM_ID] (or creates it and links)
• org-store-link & insert - OK - links to [denote:ID::CUSTOM_ID] (or creates it and links)

for (setq denote-org-store-link-to-heading nil)

• denote-org-extras-link-heading - OK - links to [denote:ID::CUSTOM_ID] (or creates it and links)
• org-store-link & insert - OK - links always to [denote:ID]

Also, please tell me if what is being created is an ID or CUSTOM_ID. The latter comes from Denote, but the former is not our doing.

CUSTOM_ID is created.

[protesilaos]

From: jarofromel @.***> Date: Wed, 18 Sep 2024 08:03:32 -0700

I've tested on the fresh install of emacs 29.4, mixed results I would say:

[... 19 lines elided]

Thank you for taking the time to do this!

I will review the NOK parts tomorrow morning. Then I will update you on my status.

Also, please tell me if what is being created is an ID or CUSTOM_ID. The latter comes from Denote, but the former is not our doing.

CUSTOM_ID is created.

This is not good. Maybe I will find what it is while working on the NOKs you mentioned.

More to follow!

-- Protesilaos Stavrou https://protesilaos.com

[jarofromel]

Just to be sure this is the format of CUSTOM_ID created in the situation (setq denote-org-store-link-to-heading 'id)

:PROPERTIES:
:CUSTOM_ID: h:fda23a69-c36b-4c98-b332-39e8f5d6b368
:END

I wasn't sure I full understood your worry about it so here is the real-life example.

[protesilaos]

Just to note that I got some unexpected task yesterday and could not work on this. I will probably be able to review it tomorrow.

[protesilaos]

Hello again! I am working on this now and made some changes.

for (setq denote-org-store-link-to-heading 'context):

• denote-org-extras-link-heading
  • heading has CUSTOM_ID - OK - links to [denote:ID::CUSTOM_ID]
  • heading without CUSTOM_ID - NOK - goes thru selection process (select file then heading) but after confirmation do nothing

I have fixed this now. A context link should be inserted at point. The old behaviour is retained.

• org-store-link & insert
  • heading has CUSTOM_ID - NOK - link created but as [denote:ID::context]
  • heading without CUSTOM_ID - OK - as expected [denote:ID::context]

I reviewed this. I think the CUSTOM_ID should take priority in this case. There is a PROPERTIES drawer, so my assumption is that the user is opting in to it---if they do not want it, they can just delete it.

for (setq denote-org-store-link-to-heading 'id)

• denote-org-extras-link-heading - OK - links to [denote:ID::CUSTOM_ID] (or creates it and links)
• org-store-link & insert - OK - links to [denote:ID::CUSTOM_ID] (or creates it and links)

for (setq denote-org-store-link-to-heading nil)

• denote-org-extras-link-heading - OK - links to [denote:ID::CUSTOM_ID] (or creates it and links)
• org-store-link & insert - OK - links always to [denote:ID]

Okay, we are good with these.

---

Did you also try the heading backlinks? I just fixed a bug there. They should be good now.

[jarofromel]

org-store-link & insert

• heading has CUSTOM_ID - NOK - link created but as [denote:ID::context]
• heading without CUSTOM_ID - OK - as expected [denote:ID::context]

I reviewed this. I think the CUSTOM_ID should take priority in this case. There is a PROPERTIES drawer, so my assumption is that the user is opting in to it---if they do not want it, they can just delete it.

NOK case still creates the link in the form [denote:ID::context]. E.g. there is a PROPERTIES drawer but the link is in the context form (CUSTOM_ID does not take priority).

---

Did you also try the heading backlinks? I just fixed a bug there. They should be good now.

Yeah, the backlinks seem ok to me. I tried denote-backlinks (with denote-backlinks-toogle-context switch) and denote-org-extras-backlinks-for-heading with various combination of links. There is one thing which bothers me with denote-org-extras-backlinks-for-heading you are right I think -> If the heading has CUSTOM_ID and there are both CUSTOM_ID and context links to the heading (in one file) only the CUSTOM_ID links are shown.

[protesilaos]

From: jarofromel @.***> Date: Sun, 22 Sep 2024 15:12:17 -0700

org-store-link & insert

• heading has CUSTOM_ID - NOK - link created but as [denote:ID::context]
• heading without CUSTOM_ID - OK - as expected [denote:ID::context]

I reviewed this. I think the CUSTOM_ID should take priority in this case. There is a PROPERTIES drawer, so my assumption is that the user is opting in to it---if they do not want it, they can just delete it.

NOK case still creates the link in the form [denote:ID::context]. E.g. there is a PROPERTIES drawer but the link is in the context form (CUSTOM_ID does not take priority).

Oh, right! I made a change now.

---

Did you also try the heading backlinks? I just fixed a bug there. They should be good now.

Yeah, the backlinks seem ok to me.

I tried denote-backlinks (with denote-backlinks-toogle-context switch) and denote-org-extras-backlinks-for-heading with various combination of links. There is one thing which bothers me with denote-org-extras-backlinks-for-heading you are right I think -> If the heading has CUSTOM_ID and there are both CUSTOM_ID and context links to the heading (in one file) only the CUSTOM_ID links are shown.

I made some changes to cover this use-case as well. Now we should return both CUSTOM_ID and context links when (setq denote-org-store-link-to-heading 'context)

But I did not make it return context backlinks at all times, because I think that is easy to return false positives. Plus, setting the user option 'denote-org-store-link-to-heading' to 'context' is the opt in we need to relax this criterion (so matches can have false positives).

-- Protesilaos Stavrou https://protesilaos.com

[jarofromel]

Link creation works as expected - I checked it in all cases as described above, shortly:

• link creation with denote-org-extras-link-to-heading or org-store&insert-link
• with nil/'id/'context in setq denote-org-store-link-to-heading

---

Now the backlinks:

After rethinking I got the same opinion as you - when the option 'context governs the linking facility should be more "liberal". So as the link creation is possible in two forms (use CUSTOM_ID if exists, otherwise use the context) the backlink should be the same -> return both forms of links. For the other option 'id' should be more strict and return only CUSTOM_ID.

And the implementation: For 'context' option it works as decribed above , both forms are returned with denote-org-extras-backlinks-for-heading or denote-backlinks (with denote-backlinks-toogle-context switch).

But for 'id option :

• denote-org-extras-backlinks-for-heading
  • returns the message denote-link--prepare-backlinks: No backlinks for query ‘yyyymmddTHHMMSS::#heading name’ if the heading has CUSTOM_ID - this is NOK
  • Otherwise it returns the message No way to get link to a heading at point ...... for the heading without CUSTOM_ID -> this is OK
• and denote-backlinks (with denote-backlinks-toogle-context switch) always returns a minibuffer with both link forms
  • in this case I am not sure what is right ... only CUSTOM_ID should be visible?

[protesilaos]

From: jarofromel @.***> Date: Mon, 23 Sep 2024 09:42:16 -0700

Link creation works as expected - I checked it in all cases as described above, shortly:

• link creation with denote-org-extras-link-to-heading or org-store&insert-link
• with nil/'id/'context in setq denote-org-store-link-to-heading

Very well!

---

Now the backlinks:

After rethinking I got the same opinion as you - when the option 'context governs the linking facility should be more "liberal". So as the link creation is possible in two forms (use CUSTOM_ID if exists, otherwise use the context) the backlink should be the same -> return both forms of links. For the other option 'id' should be more strict and return only CUSTOM_ID.

Okay, so keep this policy.

And the implementation:

For 'context' option it works as decribed above , both forms are returned with denote-org-extras-backlinks-for-heading or denote-backlinks (with denote-backlinks-toogle-context switch).

Fine!

But for 'id option :

• denote-org-extras-backlinks-for-heading
  • returns the message denote-link--prepare-backlinks: No backlinks for query ‘yyyymmddTHHMMSS::#heading name’ if the heading has CUSTOM_ID - this is NOK

Thank you! I just fixed this.

• Otherwise it returns the message No way to get link to a heading at point ...... for the heading without CUSTOM_ID -> this is OK

Good!

• and denote-backlinks (with denote-backlinks-toogle-context switch) always returns a minibuffer with both link forms
  • in this case I am not sure what is right ... only CUSTOM_ID should be visible?

This is expected because in that scenario we search for something like "denote:ID". This is going to match every kind of link. I think we should keep it this way, because users can modify those links to include some search query (the same way the 'file:' link type works).

If we were to control this, we would inadvertently limit what 'denote:' links can look like.

-- Protesilaos Stavrou https://protesilaos.com

[jarofromel]

I've just checked that last fix you did and it works of course.

Thanks a lot for your time and taking the implementation of this feature. It helps me a lot and I hope it will be useful for other users as well.

[protesilaos]

From: jarofromel @.***> Date: Thu, 26 Sep 2024 02:46:18 -0700

I've just checked that last fix you did and it works of course.

Very well!

Thanks a lot for your time and taking the implementation of this feature. It helps me a lot and I hope it will be usable also for useful for other users as well.

You are welcome! And thank you for taking the time to test it.

I did not have much time these past few days to work on my packages. I will finalise this on Sunday, if all goes well, and then merge it into the main branch.

-- Protesilaos Stavrou https://protesilaos.com

[protesilaos]

I just merged the branch into main. We can safely close this now. Thanks again for your involvement!