[FR] Automatically standardize links and export notes in vaults configured for it

Please confirm

[x] I am running the latest version of this plugin

Is your feature request related to a problem? Please describe. One of my vaults is going to be amassing information that I want to share with other people, preferably hosted on a website after syncing to a server. This would be easiest with a simple conversion as could be achieved through tools like pandoc. However, since Telekasten is intended for use with markdown with nonstandard links, most tools cannot process things properly.

Describe the solution you'd like Telekasten could use an external tool like pandoc and automatically trigger an export on writing a file if the feature is enabled. It should ideally be configured on a per vault basis, and most users would probably prefer it defaulted to off. Links would be properly identified as external or internal to the vault's content, and internal links would have their extension changed and optionally be prefixed with a string to supply, e.g., protocol, domain name, and base directory, so an internal link to subdir/entry.md can be converted to https://example.com/content/subdir/entry.html. I believe no other content in documents would need to be altered for conversion. An extra option could be provided to give users the opportunity to set any extra command line options they may need when the command is passed to the external tool.

Describe alternatives you've considered Even if markdown-related plugins may have the ability to automate exports, Telekasten's lack of support for markdown links forces users to use nonstandard links that other tools cannot automatically parse. As a result, some specialized work must be done to convert these to a standardized format that can be processed with other external tools.

This is a duplicate of https://github.com/renerocksai/telekasten.nvim/issues/126, so I am closing this issue.

I agree that markdown link is something that should be supported. But this calls for a complete rewrite of lots of telekasten inner functions and neither rene nor I have time to do that. Anyone that wish to develop this is more than welcome to do so and I will gladly review the PR.

That being said, Telekasten is intended to be a note-taking plugin for nvim. Interfacing with whatever other software is out of the scope and will not be considered (at least in any foreseeable future). There are as many workflows as there are users, trying to support everything without discrimination leads to a very messy code base that is hard to maintain (which is basically the current situation).

In this specific case, what you could do while waiting for a proper markdown link support (besides trying to fix that in telekasten directly) is to create a small script (publish_vault.sh) that will:

make a temporary copy of your vault
use sed with some clever regex to replace the links in the copy automatically
run pandoc with the appropriate options
Update everything to your server (using rsync)

This does not look very complicated. You can post the gist to such script in reply to this issue so people can refer to it if they have the same workflow.

I disagree that this is a duplicate of #126. We're asking for two totally different things. #126 wants Telekasten to support markdown links. This issue is about the fact that, as far as I can tell, these wiki links are here to stay because even if #126 is completed, it does not say that wiki links should be discontinued, so this blend of formats will forever remain an issue that is to at least some degree unique to Telekasten and applicable to some subset of its users.

I think it's reasonable to handle within Telekasten itself for a few reasons.

It doesn't really add much code complexity since Telekasten already has most of the required functionality. Telekasten can already identify a link and find its target, so these features can be leveraged to complete the task. Once you've found the link and identified the target, all that's really left to do is convert it to markdown and slap a prefix on it. It may require a little refactoring to split functionality into pieces reusable for this new purpose, but with a refactor already in discussion and maybe in progress, this doesn't seem like an unreasonable or particularly burdensome requirement.
When #126 is eventually completed, this functionality can be leveraged to help existing users automatically convert all the links in their vaults to markdown links, which would be immensely useful for users who have been settling for wiki links since markdown ones are unavailable, and it would be very important if there is ever a decision to phase out wiki links.
It's a problem Telekasten itself created by breaking from the standard it knowingly mostly-adopted, so it should be rectified by Telekasten itself, too. There are dozens of tools for working with known, commonly used standards, and Telekasten specifically ruined interoperability with these tools by tweaking an extremely common one just enough to break that interoperability in a particularly critical way.
I think it's highly relevant to Telekasten's stated purpose as "for working with a text-based, markdown zettelkasten/wiki". Zettelkastens are largely personal, but wikis are less explicitly so, and if we step back just a bit and examine the core of what Telekasten is trying to be, it is essentially a way to build a repository of knowledge leveraging commonly accepted standards for formatting that information (markdown) and structuring it (wiki/zettel). It's already a great tool for building a repository of knowledge that should be shared, but it explicitly hinders the sharing process with nonstandard wiki links, and even markdown links could use a small tweak to achieve that. I don't think that preferring to use a tool like neovim with Telekasten should be presenting additional obstacles to this goal when it is so easy to remove them as discussed in my first point.

This is something I would personally like to help implement. I would prefer to help with #126 first since I'd personally rather fully adhere to the standard of markdown, and it would help foster understanding of the code base and help identify code that can be refactored for use here. These are the two biggest features I find lacking in Telekasten compared to other plugins attempting to fill a similar niche such as vim.wiki, wiki.vim, neorg, etc., especially the markdown links.

If you are adamant about not interfacing with external software, I could also see this being facilitated by providing users with an extra option for vault configuration for a user-defined lua function to be executed when opening a file in the vault. This would allow users to set their own things to trigger on buffer write or whatever their workflow dictates. That said, I don't particularly agree with that path since it forces users to continually reinvent the functionality for link conversion, and I am not keen to personally work on this approach to the issue.

Ok I think I misunderstood what you were asking originally. My bad. The functionality in iteself is useful. The fact that it is needed for pandoc is irrelevant. Here are my comments on your reply:

It doesn't really add much code complexity since Telekasten already has most of the required functionality.

Agree. Once Markdown links are supported, and once everything is cleanly refactored, this may be just consist in a small functions or two. But as of today, none of that are currently being worked on. So it is a major task.

this functionality can be leveraged to help existing users automatically convert all the links in their vaults to markdown links,

Converting wikilinks to markdown links is indeed a mandatory implementation right after the markdown links are supported.

It's a problem Telekasten itself created by breaking from the standard it knowingly mostly-adopted, so it should be rectified by Telekasten itself, too.

I see your point, but wikilinks are also a standard, just not the same as the regular markdown one. Telekasten did not just come with it out of nowhere.

I think it's highly relevant to Telekasten's stated purpose as "for working with a text-based, markdown zettelkasten/wiki". Zettelkastens are largely personal, but wikis are less explicitly so, and if we step back just a bit and examine the core of what Telekasten is trying to be, it is essentially a way to build a repository of knowledge leveraging commonly accepted standards for formatting that information (markdown) and structuring it (wiki/zettel). It's already a great tool for building a repository of knowledge that should be shared, but it explicitly hinders the sharing process with nonstandard wiki links, and even markdown links could use a small tweak to achieve that. I don't think that preferring to use a tool like neovim with Telekasten should be presenting additional obstacles to this goal when it is so easy to remove them as discussed in my first point.

This is your interpretation of it. Once again, "a small tweak to achieve what I want" is what everyone will say when a new tool comes round the block and they want interoperability. Eventually this will result in a total mess impossible to maintain. So this argument in itself is irrelevant, although as stated above, I agree that converting to markdown links is something desirable in itself for telekasten.

This is something I would personally like to help implement. I would prefer to help with #126 first since I'd personally rather fully adhere to the standard of markdown, and it would help foster understanding of the code base and help identify code that can be refactored for use here.

AWESOME! Please do. I agree that tackling #126 first is the way to go. Then adding the automatic conversion of wikilinks to markdown links, and finally whatever 2/3 functions are required to be fully compliant with pandoc and close this current issue. This should be 3 separate PR to make review easier. If you wish, you can also start looking into the refactoring branch and cleanup a bit the mess before tackling any of that. While I do not have time to do the heavy lifting myself, I can definitely find a few couples of hours a week to review your progress and discuss implementation with you.

If you are adamant about not interfacing with external software.

What I fear is jumping blindly into adding new niche features that will eventually prove to be a hell to maintain. A good example of that is the telekasten filetype that was added (at my request) and seems to be at the origin of many bugs today. So I want to be very careful about what it means to support an external tool. Is it an actual feature that will benefit everyone, or something that will add 500 lines of code just for 3 people.

I could also see this being facilitated by providing users with an extra option for vault configuration for a user-defined lua function to be executed when opening a file in the vault.

I feel that Telekasten has already plenty of options. This also induces difficulty in maintaining it. So I would rather avoid adding too many whenever possible. As you pointed, this is not an ideal option anyway.

So in conclusion, I will reopen this issue as this functionality is useful and much needed (once #126) is done. To move forward, I suggest

(Continue working on the refactoring branch)
Solve #126
Add way to convert wikilinks into regular markdown links
Add way to convert markdown links into wikilinks (only for notes, not images or external)

Ideally the same function can be used for 3. and 4., just put the old and new regex in arguments. This way we could convert anything into anything else.

It's a problem Telekasten itself created by breaking from the standard it knowingly mostly-adopted, so it should be rectified by Telekasten itself, too.

I see your point, but wikilinks are also a standard, just not the same as the regular markdown one. Telekasten did not just come with it out of nowhere.

Yes, but the problem arises from mixing standards. I can see the utility of the decision in an initial version because wiki style links provide an unambiguous distinction between internal wiki links and external markdown links, which removes the need to implement any logic to distinguish them if they were exclusively in one style or the other. As a consequence, however, Telekasten leans on two standards but is ultimately compliant with none, and none of the many tools that could cover wiki or markdown styles on their own know how to handle Telekasten's style.

Then adding the automatic conversion of wikilinks to markdown links, and finally whatever 2/3 functions are required to be fully compliant with pandoc and close this current issue.

I don't particularly care if pandoc or some other tool is used. The main issue is that none of them are really viable because of the deviation from the standards it uses. My limited knowledge suggests pandoc is the most popular and powerful tool for the job, so it's my preference, but I'd be fine with another. Could alternatively be implemented as an option so users could pick their own export tool with maybe a table of three values, command, flags for before the file name is provided, and flags for after the file name is provided. As long as Telekasten picks a single standard to abide by, it doesn't really matter what tool is used to export.

I'll clone and dig around in the refactoring branch some time soon and see what I can do in there. I think that'll be a good way to get a good understanding of the code and solve some of my own future problems along the way by splitting things up how I'll be needed them soon after.

Discussions like this are wasting everyone's time given the circumstances. Fact of the matter is, unless you send a PR this will not get implemented soon. Both Thomas and I are way overbooked so I don't see a quicker way than someone else coming up with a PR. The PR approach itself is also not super straight forward because of the ongoing refactoring work. Just reiterating how non-standards-conformant Telekasten is not helping anyone.

There have come a lot of Zettelkasten-affine tools out in the past few years, and they pretty much all stuck to wiki link notation. Some with IDs, some without IDs, but wiki links are pretty much the standard here, like it or not.

I quite like the distinction between internal wiki links and external markdown links.

The way I did it in sublime_zk and sublimeless_zk and how plugins for other well-known software like Obsidian deal with "sharing" is: providing an export feature. When attempting an export, you'll soon also run into relative links to images problems, copying around assets, etc. It is not rocket science, and certainly can be done. But I would rather have someone with experience do it than someone who just thinks how everything ought to work.

Telekasten picking a standard.... It started out like this. There was only one way, and the world was simple. Then everybody brought along their extra wishes, tag formats, link formats, etc. - and things started to become messy. The moment Telekasten "agrees" on one standard, we'll have dozens of complaints of people whose archive just broke demanding an urgent fix.

I am not even sold on it necessarily having to be a Telekasten feature. It's basically: generate static HTML from a wiki markdown base, and as such not limited to use in Telekasten. Any rather generic plugin with a suitable generator would do.

Anyways, we can write lengthy arguments here but they won't bring us closer to a solution. Sorry about that. It's just the reality of an open-source-project that thousands of people use, hundreds demand to be changed, and two doing the work.

Sorry I don't have time for anything more here, not even making it sound more friendly. I have a deadline to meet today.

Discussions like this are wasting everyone's time given the circumstances. Fact of the matter is, unless you send a PR this will not get implemented soon. Both Thomas and I are way overbooked so I don't see a quicker way than someone else coming up with a PR. The PR approach itself is also not super straight forward because of the ongoing refactoring work. Just reiterating how non-standards-conformant Telekasten is not helping anyone.

I disagree entirely. I'd never spend the time to dive into an unfamiliar code base and develop features for a PR without trying to start a discussion. As has come up in this very discussion, not every idea for features aligns with the core developers' vision for the project, and who wants to sink all that time just for it to go unappreciated? As it is, this has directly lead to an amateur developer with an interest in helping gaining insight into how best to help both for the longevity of the project and to see my own desired features implemented in a way that keeps everyone happy.

I quite like the distinction between internal wiki links and external markdown links.

And I have no issue with keeping them. Adding the functionality I want could provide a way to also phase wiki links out if the project saw that as desirable, but I'm not surprised that some users find the distinction useful, and I don't see the need to change that. I just think that an intentional break from standards should be accompanied by a way to conform to them for interoperability with other tools. In this case, this means being able to use internal markdown links if the user chooses and being able to convert links if the user has chosen not to. By the point that that is accomplished, it's trivial to make some minor adjustments to internal links as part of the export process, so it comes as a nice, easy bonus.

I am not even sold on it necessarily having to be a Telekasten feature. It's basically: generate static HTML from a wiki markdown base, and as such not limited to use in Telekasten. Any rather generic plugin with a suitable generator would do.

If Telekasten had chosen a single standard to use, I would be much more inclined to agree. The fact that it has not means that any external tool must custom tailor itself to Telekasten. It seems much easier and more reasonable for Telekasten to standardize its content for export so that it conforms to the ecosystem rather than forcing the ecosystem to find ways to adapt to Telekasten. It doesn't even impact anybody's preexisting workflows to do so. Users would be free to continue using the format they're already using, and those who need to export would face no complications regardless of the choice they made.

I'm going to try to start working on things today, and the list and ordering @lambtho12 provided seems like the best way to do so, so I think I'm going to start by figuring out why the refactoring branch can't be automatically merged and go from there.

I disagree, if I understood you correctly. Converting links should be part of the export process. As all other ZK-affine tools I've ever used use wiki links for links to notes yet still allow for md links to external resources, it's not unlikely that external exporters exist, as the sharing topic comes up with all of them. I've written exporters for the 2 ZK tools I've authored before and integrated them. Obsidian has a plugin that works similarly. I favor the idea that not every note-taking solution should need to reinvent exporting markdown notes linked together by wiki links. It should focus on getting its core part right. Markdown notes are future-proof. They will keep working even when you switch tools. The value of an external exporting tool with nice customization options becomes apparent the minute you switch tools. Imagine the new tool is perfect but doesn't care about exporting. Or it has one but then your export would look and feel differently....

You still seem to think that this would mean that other tools would need to tailor themselves to Telekasten. That is simply not true, as there are so many tools out there that use the same wiki links. I developed Telekasten because our company introduced Obsidian, and I wanted to stay in neovim but still be able to work with Obsidian vaults. An exporter for Telekasten would also work with Obsidian vaults and many others - and vice versa.

Wiki links are cool because they don't need a fancy editor / viewer to hide the redundant parts. That's why so many tools use them. Converting from markdown format a to format b just before exporting seems like an unnecessary step to me.

IMHO the best option is a variant of a static site generator. No matter if you use The Archive, Obsidian, VScode, Telekasten, ... - you just export. Without having to convert [[this]] into [that](that.md) beforehand.

A Telekasten-specific export, while certainly adding value, would seem like an unnecessary limitation to me, leaving out all users of other tools unnecessarily.

But I guess no argument is going to convince you, so please go ahead and do what feels right for you. It's not like you would need permission anyway. I will continue to not have time for more than the odd rant on an issue, unfortunately, and certainly wouldn't block a valuable addition to Telekasten, no matter whether I find it optimal or a local optimum.

BTW: I see no harm in usig markdown links internally. I just find them unnecessarily cumbersome for internal links. The wiki links are a feature, not a bug IMHO. But adding the complexity to support multiple link formats would have meant spending time on a feature I didn't care about, having less time to spend on features I did care about or to allocate to other important projects.

@lambtho12 has done a really awesome job, and I hope that after his refactoring work, it should be much easier to make local changes without unintended side-effects. I strongly recommend, like you suggested, basing your intended changes on his refactored state of the codebase.

nvim-telekasten / telekasten.nvim

[FR] Automatically standardize links and export notes in vaults configured for it #294