Broken links in some .org files

[xfq]

The TOC links in the following files are broken (404):

• https://github.com/syl20bnr/spacemacs/blob/master/doc/DOCUMENTATION.org
• https://github.com/syl20bnr/spacemacs/blob/master/doc/FAQ.org
• https://github.com/syl20bnr/spacemacs/blob/master/doc/CONVENTIONS.org
• https://github.com/syl20bnr/spacemacs/blob/master/doc/LAYERS.org
• https://github.com/syl20bnr/spacemacs/blob/master/doc/QUICK_START.org
• https://github.com/syl20bnr/spacemacs/blob/master/doc/VIMUSERS.org
• https://github.com/syl20bnr/spacemacs/blob/master/layers/LAYERS.org
• layers/**/README.org (I haven't tried all of them)

I think the broken links were introduced in 13c5b1d (Nov 2, 2015).

PS: the TOC links in https://github.com/syl20bnr/spacemacs/blob/master/CONTRIBUTING.org are not broken.

---

Update: Not only TOC links, but most (if not all) internal links.

[syl20bnr]

This is a known issue and we have yet to find a solution for this.

[TheBB]

I thought we landed on wontfix for this, and to have http://spacemacs.org be the "official" way to view documentation online.

[StreakyCobra]

I thought we landed on wontfix for this, and to have http://spacemacs.org be the "official" way to view documentation online.

We landed on this yes, but imho this is not a viable long-term solution. Search engines are indexing those pages, some users read the documentation on github, some users use the github search box to find information in the project, etc.

[robbyoconnor]

Add the wont fix label :D

[StreakyCobra]

Nah, we need a better solution. Discussing with @TheBB recently we were seeing a potential solution to have links working everywhere, but we are both kind of busy those days.

[louwers]

Add a custom id to every header and link to those like so:

* Spacemacs documentation                                   :TOC_4_org:noexport:
 - [[#core-pillars][Core Pillars]]
   - [[#mnemonic][Mnemonic]]
   - [[#discoverable][Discoverable]]
   - [[#consistent][Consistent]]
   - [[#crowd-configured][Crowd-Configured]]
 - [[#highlighted-feature][Highlighted feature]]
 - [[#screenshots][Screenshots]]
 - [[#who-can-benefit-from-this][Who can benefit from this?]]

Then it works from org-mode (export) and on GitHub.

It is not optimal I have to admit.

[TheBB]

Yeah, those drawers become an eyesore when reading from within Emacs.

[louwers]

By the way, if you want people to read the documentation on http://spacemacs.org you could at least link to that from the README.

[xfq]

@louwers I submitted a PR (#4737) for that just now.

[JAremko]

Came across this issues. Absolutely had to use my favorite Emacs feature #4811

[TheBB]

Never mind exporting. I can't even find a way to make a link to a heading in another file work well across github and Emacs.

I really, really, really don't want to spend my time working on this. Are we so sure this is a blocker? I can't see any solutions that are not going to be very flimsy.

[TheBB]

And no, I'm pretty sure the links we had that pointed to specific headings in other files were already broken in the previous versions.

[JAremko]

So this is my plan how to fix it:

• [X] Try stupid stuff
• [X] Try even more stupid stuff
• [X] Understand that the stupid stuff is stupid and try to come up with something less stupid
• [X] Come up with a bold plan:
  Instead of trying to shoehorn some kind of insanity into the org files or Spacemacs to make .org stuff almost work both on GitHub and in the Spacemacs - with all kind of ugliness like # everywhere or broken links here and there - we can simply fix the org files for GitHub when they are on GitHub and for Spacemacs when they are in Spacemacs! With all kind of benefits.
  TODO:
  • [x] Make a function for formatting org files for GitHub #5342
  • [ ] Make a function for formatting org files for Spacemacs i.e. make them normal org files. And hook it up to the layer install process. #5345 (WIP)
  • [ ] Add Travis test to make sure that PR's README.org files are properly formatted and will look normal on GitHub.
  • [ ] Add Contrib-layer that will help in creating new layers, testing them, formatting, and creating PR that follows the guidelines. bonus level!

@syl20bnr I need a feedback before I spent all of my free time on this :smile:

[TheBB]

@JAremko We don't have any control over github. How do you propose to have github run your githubify function? Do you want to have a "rendered for github" copy of the docs in the Spacemacs repo?

[JAremko]

@TheBB Here is how it works:
Contributor can run it before submitting a layer. To make it even simpler I want to build a pipeline that will allow creation of well formed PRs right from the Spacemacs. It will warn is something is malformed and run checks and formatters before getting to the PR part. That the GitHub stuff.

Now to the Spacemacs installation process: When Spacemacs updates or installs a layer it will check is corresponding README.org canonical or not. And if it's githubified - it will rebuild it to be vanilla org file.

[JAremko]

@TheBB Ah. Also I forgot to mention that I'm planing to add extra test on PR's for checking README.org files. It should prevent most of #5253

So. What is your opinion?

I'll try to not commit anything before we come to some kind of conclusion :smile:

[TheBB]

The actual README.org file on disk, what format will it be in? To render properly on github it has to be in github format, right?

When Spacemacs updates or installs layers it will check is corresponding README.org canonical or not. And if it's githubified - it will rebuild it to be vanilla org file.

You can't save org-format READMEs to disk though. They have to be in github format in the repo and if you overwrite them the repo will be dirty, and that's a no-go. Any changes you want to make to the buffer must be when you view the file from within Spacemacs. You can't save it in its modified state, so there's no use doing this at the install or update stage. Right?

toc-org already solves part of the problem. It generates a TOC in github format (actually we use it right now to generate a TOC in org format, but we can switch to github format) and it automatically translates links on-the-fly so that org can use them. The problem is that it can only do this for links within the same file. If we can extend it to work across files, we have a workable solution. Unfortunately this is not simple at all.

[JAremko]

The actual README.org file on disk, what format will it be in? To render properly on github it has to be in github format, right?

Right.

You can't save org-format READMEs to disk though. They have to be in github format in the repo and if you overwrite them the repo will be dirty, and that's a no-go. Any changes you want to make to the buffer must be when you view the file from within Spacemacs. You can't save it in its modified state, so there's no use doing this at the install or update stage. Right?

So far it looks like it won't be any problem to make githubify and ungithubify fully cancel each other. So you can revert the README.org file to the original state. But sure, we simply can run ungithubify in org-mode hook.

toc-org already solves part of the problem. It generates a TOC in github format (actually we use it right now to generate a TOC in org format, but we can switch to github format) and it automatically translates links on-the-fly so that org can use them. The problem is that it can only do this for links within the same file. If we can extend it to work across files, we have a workable solution. Unfortunately this is not simple at all.

I know. but it's ORG based and I can't fiddle with it. So it's not perfect for my shenanigans :smile: Also it should be possible to convert links from "http://" form to org form if the org file is in local Spacemacs repo, by rewriting them relative to its root.

[JAremko]

If we can extend it to work across files

Btw @TheBB do you have an example of such link in Spacemacs docs? How it is now and what we want it to be. So I will have something to run tests on. Thanks!

[JAremko]

I decided to implement both githubifying and ungithubifying as a single function with a flag so it will be straightforward to Undo/Redo the changes so I closed #5345.

How it can be used:

Publishing a layer:

1. run (spacemacs/githubify-orgbuf t) on the layer's README.org - format README.org to work properly with GitHub
2. save file
3. publish to gihtub

Installing Spacemacs:

1. install layers
2. open README.org files and run (spacemacs/githubify-orgbuf nil) - make local README.org vanilla org with the org links
3. save updated files

Updating Spacemacs:

1. open README.org files and apply (spacemacs/githubify-orgbuf t) - make local README.org the same as in the GitHub repo.
2. save them
3. run update
4. apply (spacemacs/githubify-orgbuf nil) - Turn local README.org into vanilla org again.

The last one is trickier. So it might be better to simply modify README.org buffer's when they are opened. Or even make it a part of #5222 The downside is that the local README.org files sill will be "messy" while working properly in the "view-mode" and it potentially might be problematic with some tools or if someone wants to use them outside of Spacemacs.

• it's shouldn't be hard to transform web links into the local relative links when it's appropriate. For example, in the githubified org file a link to a heading in some other org file could look like https://github.com/syl20bnr/spacemacs/blob/master/layers/chinese/README.org#install ungithubify will notice that it's a link to some place in syl20bnr/spacemacs/ repo so it will rebuild it into something like this file:/home/jare/.emacs.d/layers/chinese/README.org::*install or a relative path to the file that contains the link.
• We can replace some org elements that GitHub doesn't render properly with a Unicode symbols or icons in the githubified version.

I'll wait for the feedback on the plan.

[StreakyCobra]

Having a file on my computer that is a different version of what is on Github is nonsense for me. I would for sure disable such feature on my side.

[JAremko]

@StreakyCobra Not sure if the end user even notice that they are different. But I agree with you - it's a ton of extra complexity to solve really narrow use-case :+1: Also if someone really wants to have the cleaned version - the person can simply save the buffer to a file. It might be better to hook up ungithubify with the view-mode.

So how about this:

Publishing a layer:

1. run (spacemacs/githubify-orgbuf t) on the layer's README.org - format README.org to work properly with GitHub probably managed via contributor-layer if I make one.
2. publish to gihtub

Viewing some README.org file for a layer in view-mode:

1. User opens the file with SPC h SPC - it enables the view-mode
2. The decorator from #5222 hides meta and runs (spacemacs/githubify-orgbuf nil) to rebuild TOC, remake web links to relative local org links, undo (spacemacs/githubify-orgbuf t) replacements like, for example, org checkbox fix ☑ ☐ etc.

Is it OK with you @TheBB @StreakyCobra ?

[syl20bnr]

I hope I understood correctly the issue, bare with me if the following proposal does not fix the real issue.

If we want the links to work on GitHub then we have to push files that GitHub renderer understands. OTOH we have full power in the Emacs ecosystem so here is what I suggest to do: 1- We revert toc-org to GitHub format to make GitHub happy 2- We advise the functions used to open a link and we detect the link format to replace the argument in the hope that org will understand it. (best is to advise at the org level, not too deep). 3- unit tests with good use case coverage are mandatory for this contribution.

[syl20bnr]

For the website part we have full freedom since we export the files so I don't expect new issues to arise for this.

[TheBB]

Even better, you can use org-link-translation-function (which is what toc-org does). However it's tricky to get it to work properly.

[TheBB]

The website won't immediately break but eventually we have to figure out what to do there too.

[JAremko]

First of all we need a test for toc-org It might be something that we won't deploy - just to evaluate the proposals. I think I'll hack something with bash and curl that might even work in Travis or do something that isn't fit for the production. If toc-org fixes all of TOC links (including Chinese) I think we can scrap githubify But I have some doubts given its char list - if I'm correct it shouldn't work with the Chinese TOC. But also mine has high chance to not cover all the cases :smile: + it's clearly inferior totoc-org

But also I think we can do better than just a link fix. We can fix other stuff like checkboxes. But then we'll need to revert it for Spacemacs. While it's not the subject of the issue - it will affect how we approach it. If we go for the "fix 'Em all" It can be more straightforward to perform all of the magic in a single place. Might be even before org-mode is loaded - so the buffer will look like an absolutely normal org file without link "runtime" magic and hookery so users and tools will be happy. Also it should be more performance friendly solution. Also it should be much easier to test. We can run regexps on the transformed buffers, compare them. But it's hard to test hooks.

So. What's the plan?

[JAremko]

It looks like GItHub uses org-ruby internally. We might be able to use it or markup for testing and it should contain the full char list for the removal.

[syl20bnr]

Let's focus on links, for now I don't see how we can use check boxes or other controls in documentation pages, we won't do any interactive stuff in them. For the tests we only need unit tests (i.e. testing translation of urls to expected format) so curl is not needed.

@JAremko What do you mean by supporting chinese links ? Seems that the TOC in the chinese layer is working, is there a bug when using the markdown format with org-toc ?

[JAremko]

@syl20bnr the last character "）" in Set monospaced font size(设置等宽字体） could be missed by toc-org so the TOC link will contain it and will be broken. Or if it had №

[JAremko]

I don't see how we can use check boxes

For the TODO mb? :confused:

[syl20bnr]

For the TODOs we have GitHub issues and wiki ;-)

[JAremko]

I dug deep and long and all over the place to discover that GitHub actually uses Pretty simple rule - it's kinda embarrassing after what we used :smile:

For some reason I can't make replace-regexp-in-string work with "[^\\w\\- ]" so I just reported it snosov1/toc-org#12 I think if we solve this issue toc-org will work just fine for GitHub. Then we will need to take care of links inside org mode.

[a13ph]

Proper way to handle orgmode toc on github is to contact github about that, or go directly over their head to the upstream code being used and issue/PR there, and only then contacting some octocat

[a13ph]

Or to toc-org, if that's the issue :open_mouth:

[JAremko]

@a13ph Well. GitHub can't allow all kind of symbols in URLs so the fix will be some kind of band-Aid in js that will probably move it too far away from how markdown works. Or they will have to sanitize TOC on their end i.e. changing user's content. It will be too much hassle and will take ages to push trough. Or it will be simply rejected because it's up to users to submit properly formatted content.

[JAremko]

Also the solution for handling links in Spacmacs relies on some specific stuff like that the local files will start with https://github.com/syl20bnr/spacemacs/ and then we can rewrite them (or redirect via hooks) to the local copies and open them in the org mode.

[a13ph]

I'm unsure, but isn't this relevant? https://github.com/snosov1/toc-org/issues/3

[a13ph]

Those hooks and specific stuff - it's an ugly hack, IMO. If it can be done cleanly - fine.

But i'm not sure we are the only ones with this issue and it wouldn't hurt to ask those who are responsible even if we do pursue workarounds.

Kinda strange not even opening an issue, considering where we are and who this issue is with.

[JAremko]

Kinda strange not even opening an issue, considering where we are and who this issue is with.

I'm not sure if GitHub has an issue(nor that it's not opened :smile: ). links like #foo/bar are working. And you can use them to link other documents. Sure it doesn't provide all of org utilities like text links with a search and stuff, but it is the cost of being in the web environment. It will be hard to make them do something about it.

Those hooks and specific stuff - it's an ugly hack, IMO. If it can be done cleanly - fine.

I also think that it's a bit meh. We probably could handle it without the magic by simply formatting(removing github fixes and making links local) org files when they are opened. But we can go with toc-org if syl20bnr says so.

I need to write some tests and dig into the link magic.

[JAremko]

Created a pull request snosov1/toc-org/pull/13 it should make toc-org more reliable(or, knowing me, somehow in someway broken :smile: )

[JAremko]

I'm halfway there. It will be epic PR. I only hope not too epic :smile:

Also I have found a surprisingly reasonably way to make it all work with the site.

So far it looks like we can fix all the links. Even links to headings in different files.

[a13ph]

@JAremko thanks, I think this will make reading docs much more tolerable [at least for me!] if all links work (less mental load than jumping or searching through whole file)

[JAremko]

Looks like tomorrow is the day :shipit:

[JAremko]

Here it is... #5545

[d12frosted]

Fixed with released of Spacemacs v0.200. Let us know if you still have any problems with this issue.

[alphapapa]

FYI, if this is still relevant, you may find this useful in generating tables of contents more flexibly: https://github.com/alphapapa/org-make-toc