Transclusion Support

[Feel-ix-343]

https://www.reddit.com/r/PKMS/s/ExEET5gWbo

[oblitzitate]

Yeah, I was looking at org-transclusion for such a feature, but I would love something more.

It would be cool if this had advanced transclusion features where you can transclude multiple items into one while sorting or filtering them in some way based on some metadata that each item may have.

For example, you have a bunch of tasks under different projects. Then you could transclude them all under a single heading called Today which filters in all the tasks scheduled for the current day and sorts them based on time.

I would imagine ids would probably need to be mandatory for each item. And then for any conflicts it can't handle, you could prompt the user about it.

I don't know if it's completely possible, but it would be awesome. You could make it a full blown todo app with such a feature alone.

[Feel-ix-343]

You could make it a full blown todo app with such a feature alone

Well I do something very similar to what you are saying: I link all of my tasks to [[LATER]] then get references to [[LATER]] when I want to view the tasks. This workflow will be improved when markdown-oxide supports queries -- allowing filtering by metadata like you said. Right now, I just get references to [[LATER]] then do a grep style filter if necessary. Of course, a structured filter would be better.

The followup question is how would the transclusion UI be displayed? I think the LSP hover window may be good; it could be like the obsidian reading mode. Given that the language server can send what ever it wants in the hover window, there is a lot of potential. Right now, when hovering on a reference (link) or even a definition (heading), the hover window will show a preview of the written contents in combination with backlinks. This can be made more advanced. What do you think of this option?

Of course, we could also make editor specific plugins to give the best transclusion, but this would be a lot of work and some have said it would be out of scope. I'd say: this is better than making our own text editor! but it certainly does increase the scope of the project significantly and would leave some users out of key features if they are using not fully supported editors.

[oblitzitate]

If you're striving to have the best PKMS, you would want to display the transclusion directly in the buffer, where the user can seamlessly read, navigate and edit its contents, without having to shift context. Otherwise, the user experience wouldn't be the best it could be.

I don't know the limits to which a language server could help implement such a feature, but I imagine it would at least be in charge of retrieving the original contents, updating the transclusion block, handling the edits of a transclusion block, and updating the original contents. If you need editor-specific plugins to complete such a feature, you'll have to decide if doing so is worthwhile in order to achieve the best PKMS. In my opinion, it's at least worth experimenting.

If you need examples of such a feature, there is Org-transclusion as well as Obsidian + MAKE.md plugin with make.MD: Open Flow Blocks in Selection command. Org-transclusion seems to do transclusion pretty well. Obsidian not so much as you have to select and click things to edit them. But a big shortcoming for both is that they can't have multiple transcluded elements under a single transclusion block

Being able to have multiple transcluded elements under a single transclusion block would be pretty significant as it would allow to sort them, filter them, and modify their presentation in a clean manner. If implemented properly, it'd be like having org-agenda directly in the buffer. You could transclude todo elements from project A and project B, filter by if scheduled, sort by date, modify the presentation so that their dates are shown at the beginning of the line, etc. Basically, a full blown todo app as I said before.

In terms of implementing the feature, I imagine that for each transcluded element, you'd probably need to in-line their id along with them, in order to track their positions better (for sorting and filtering).

Although, the more I speak about this, the more I think it'll take a significant amount of effort to implement. So it's up to you if you think it's worthwhile.

[Feel-ix-343]

directly in the buffer, where the user can seamlessly read, navigate and edit its contents, without having to shift context

I don't think it would be possible to have this directly in the buffer, but a user could open an Entity View (as I have been calling it in my notes) where all of those actions (even editing) would be possible. Applied to your case (which is also mine)

• One could write a query for getting all task items for the current day, while also giving sort preferences and such
• One could open the entity view for this query (as the query is the entity representing your agenda)
• In this entity view, you could view transcluded items, reorder items manually, and edit all transcluded items -- while also being able to goto definition on the transcluded item...

For my case, instead of using a query, I would use backlinks to the [[LATER]] entity, which would be included in the entity view. I add a [[LATER]] link to all of my tasks: for example - [[LATER]] fix markdown oxide bugs. To get and edit all of these, I could go the LATER.md file, open the entity view, reorder my tasks etc, ... I have my agenda!

Correct me if I am wrong, but this Entity View (in combination with queries) would fulfill what you are looking for

But still how would we implement this entity view? It can not be view only (we can not leave it at LSP hover). My ideas are...

• I make plugins for text editors; however, this by itself is not ideal as some text editors (Helix and Zed) do not have great plugin systems yet. I think the only editors this would be possible for are Neovim and VSCode. That is not too bad!
• The LSP directs users to a temp file as the entity view; the user then edits, reorders, ... items in this, the language server listens to all of these changes in detail through the LSP -- keeping track of line numbers for transcluded items -- and updates transcluded items accordingly. I expect that this would take a while to be made robust, but it surely is simple and flexible, so for editors that do not have plugin systems, this could be the experimental solution while I could make robust plugins for the supported editors.

And who knows! Maybe the temp file could be a good solution for all text editors; the LSP with hover, definition, references, rename along with a custom text-based entity view interface is grounds for extreme creativity!

[Feel-ix-343]

And if one wanted to save an entity view, it would be a simple as saving the markdown file to vault! Markdown-oxide would figure out the rest

[oblitzitate]

I don't think it would be possible to have this directly in the buffer,

It's possible in emacs with org-transclusion. When you add/open a transclusion, it simply replaces the #+transclude line with a copy of the text content it points to. Then you can edit it as if it belonged to you (while behind the scenes org-transclusion handles the editing and any inconsistencies).

I imagine this is also possible in neovim. I don't know about the other editors though.

Correct me if I am wrong, but this Entity View (in combination with queries) would fulfill what you are looking for

Similar to queries except it replaces (or adds under) the query block with an editable copy of the text contents when you open the block. Wherein I can just edit it without having to click things.

I'm not sure about Entity View. I'd have to see what you mean. From the sound of it, it doesn't sound like what I'm looking for.

I basically want to stay in the markdown file where I can contain multiple transclusion blocks and write things around or in-between them, rather than switching views. I want the experience to be seamless.

The LSP directs users to a temp file as the entity view; the user then edits, reorders, ... items in this, the language server listens to all of these changes in detail through the LSP -- keeping track of line numbers for transcluded items -- and updates transcluded items accordingly. I expect that this would take a while to be made robust, but it surely is simple and flexible, so for editors that do not have plugin systems, this could be the experimental solution while I could make robust plugins for the supported editors.

Having a temp file is interesting. If for whatever reason, you can't paste the contents directly into the buffer, then maybe a temp file could hold the transcluded contents as some sort of experimental solution.

[devzero]

Instead of temporary files, having a FUSE file system would be cross platform and editor independent. It could also implement read only views and incorporate the transcluded document with some sort of separator in the buffer itself and incorporate and propagate the changes back to the original files

[Feel-ix-343]

hm very interesting. I will certainly look into this -- thank you!

Another easy thing we could do is use codelens to give small previews. This may work especially well for embedded block links.

[Feel-ix-343]

Oh I did mean inlay hints