Rewriting for 0.5

[alphapapa]

@snosov1 Hi Sergei,

The code in org-make-toc was so ugly and inflexible that I rewrote almost all of it. It's now much simpler, more powerful, and more flexible. I would love to have your feedback on it. I've updated the docs and the example.org file.

Also, I've used the new version to redo the TOCs at https://github.com/alphapapa/emacs-package-dev-handbook. It's so much more flexible and easy to use now! It's also much faster, especially on large documents, because it doesn't parse the whole buffer with org-element anymore. (I thought about trying to rewrite it while still doing that, but it's much simpler to just work in the buffer text with a recursive function.)

@akirak @dakra @benley @mankoff @lxbarbosa @asos-stephanemaniaci I would welcome your feedback as well! It's probably easiest if you view the branch and the readme from here: https://github.com/alphapapa/org-make-toc/tree/wip/rewrite

Please let me know if you think of any missing features or improvements to be made!

[akirak]

@alphapapa I hardly remember what I can set as the value of TOC property. It would be nice if there were an interactive command to set the property with which the user can use the package without looking up the documentation.

Also, it was not obvious how I could hide an entry in the toc and the entry itself on GitHub (and from ox-exporting). Users probably want to hide entries themselves that are not visible in TOC, so those settings could be added by the new interactive command.

[alphapapa]

Hi Akira,

@alphapapa I hardly remember what I can set as the value of TOC property. It would be nice if there were an interactive command to set the property without looking up the documentation.

Thanks, that's a good idea.

Also, it was not obvious how I could hide an entry in the toc and the entry itself on GitHub (and from ox-exporting). Users probably want to hide entries themselves that are not visible in TOC, so those settings could be added by the new interactive command.

I'm not sure what you mean. Could you show me an example please?

[akirak]

I'm not sure what you mean. Could you show me an example please?

You can prevent an entry from being rendered on GitHub by setting COMMENT keyword and from being exported by adding noexport tag. I usually want to do all of these at the same time. Perhaps the interactive command should ask the user if he/she wants to add them as well when it sets TOC property to ignore.

[alphapapa]

Thanks. I'm not sure if I want to add that feature, because I think it could be confusing, a bit too much "magic." As it is, this package doesn't affect exporting or keywords, and while it has a GitHub link type, as a whole the package is not specific to GitHub. I think I'd prefer to add documentation with GitHub-related tips. What do you think?

[akirak]

I think I'd prefer to add documentation with GitHub-related tips. What do you think?

That will be sufficient. Thanks.

[snosov1]

Hey, @alphapapa !

I gave it a run today, but, unfortunately, it wasn't successful. I called eval-buffer in org-make-toc.el then went to your README.org and called M-x org-make-toc. It gave me org-make-toc: No TOC node found. and I stopped trying =) I just tried one other thing - C-c C-o on a heading in TOC and it also didn't work.

All in all - you, probably, know how I feel about toc-org alternatives. There's several features I consider must have:

1. Small cognitive and representational load
2. C-c C-o jumps
3. No "re-renderings" on unchanged TOC
4. Correct hrefs for non-unique headings

There's some other less critical but useful things, like, correct state handling (custom and special, like, TODO or COMMENT), no external dependencies, unit tests, markdown support, etc.

Anyway, as far as I'm concerned, org-make-toc lacks at least 1, 2 and 4 (not sure about 3 - couldn't test it).

For item 1, look at the comments in this very thread =). Also, your README.org file has 22 lines of code related to TOC generation - :PROPERTIES:, :TOC:, :CONTENTS: and :END: lines. The number of non-empty lines is 133 (this also includes stuff, like # before-save-hook: org-make-toc). In other words, >16% of the lines in this file is irrelevant to the content and is needed only to generate the TOC. I think it's a little bit too much =) Maybe, if you think about TOCs everyday and find it fascinating then it's ok, but for a regular person, I think, 1 functional line is more than enough =)

So, my switch is not today - again =) But the code does look nicer!

[alphapapa]

Hi Sergei,

Thanks for your detailed feedback! Let's see:

I gave it a run today, but, unfortunately, it wasn't successful. I called eval-buffer in org-make-toc.el then went to your README.org and called M-x org-make-toc. It gave me org-make-toc: No TOC node found. and I stopped trying =)

I just tried that and it works. Did you accidentally use the README.org from master instead of this wip/rewrite branch?

I just tried one other thing - C-c C-o on a heading in TOC and it also didn't work.

Yes, there is a discrepancy between the way GitHub's org-ruby Org renderer handles intra-file links and how Org itself handles them. Using links like #heading-name is necessary for GitHub, but Org expects such links to be to headings with defined CUSTOM_ID properties. I think the only way to solve that would be to assign headings CUSTOM_ID properties. I suppose that could be done automatically, but that could have undesirable side effects and would add complexity, so I haven't pursued that option yet. Ideally GitHub's renderer would be updated to handle Org-native links better, e.g. in the [[*heading name]] format that Org uses.

However, AFAICT, toc-org has the same problem. For example, I opened this example from the toc-org readme file in Org, put point on the Installation link, and pressed C-c C-o, and got the same error: org-link-search: No match for custom ID: installation.

Now I see that you have an toc-org-enable command that sets org-link-translation-function in a buffer, which is useful. But that seems to me to be orthogonal to the function of creating TOCs. That link-translation functionality could equally belong in a compatibility package that simply makes Org follow HTML-style, GitHub-style intra-file links, one which would work with both TOC-creating packages.

Small cognitive and representational load

Of course, that is a valuable feature. Did you see the new "basic TOC" steps in the readme? I feel like that's pretty straightforward. As Akira said, it would be helpful to have a command to help with configuring TOC properties, so I'll probably add one.

Besides the use of a property instead of a tag, the chief difference from toc-org's workflow is that a :CONTENTS: drawer is now used, which has significant benefits over replacing an entire entry's content. For example, you can see in the new examples.org file what an inline TOC looks like, with text before and after it. As well, when exporting Org files, specific drawer types can be excluded, so a user could exclude TOC drawers without also excluding other entry content.

Of course, the way toc-org is used to make the simplest TOC, by simply adding a :TOC: tag to a heading when the user wants to replace an entry's entire contents, is simpler than setting a :TOC: :include all property and adding a :CONTENTS: drawer, the way org-make-toc works. But besides that simplest of cases, I think org-make-toc's use of properties is more flexible and less cryptic to configure.

I also generally prefer the use of properties over tags for configuration, because tags are displayed when headings are folded, while properties are hidden when collapsed, so the TOC configuration gets out of the way. Also, properties are key/value stores, while tags require special encoding. Is :TOC_2: better than :TOC: :depth 2? Well, that depends. :)

No "re-renderings" on unchanged TOC

I guess you mean that the TOC-making command should be idempotent. I agree, and AFAIK, this is the case with org-make-toc, both in 0.5-pre and earlier versions. Am I misunderstanding you?

Correct hrefs for non-unique headings

I have indeed run into that problem in my own documents, like at https://github.com/alphapapa/emacs-package-dev-handbook. But IIUC, that can't be fixed without using ID or CUSTOM_ID properties, which would require modifying properties and adding significant complexity. Am I missing something?

correct state handling (custom and special, like, TODO or COMMENT)

I'm not sure what you mean. Would you elaborate? I haven't thought of to-do keywords as relevant to making TOC, but maybe there are improvements to be made here.

no external dependencies

I'm fond of dash.el and s.el, but they could easily be omitted if necessary. :)

unit tests

Of course. As the code matures, I'll likely add some, e.g. using the comprehensive examples.org file as test data.

markdown support

I confess that I have no interest in Markdown, given that we have Org. :) If I really need to make a Markdown document, I can export to it from an Org document.

Also, your README.org file has 22 lines of code related to TOC generation - :PROPERTIES:, :TOC:, :CONTENTS: and :END: lines. The number of non-empty lines is 133 (this also includes stuff, like # before-save-hook: org-make-toc). In other words, >16% of the lines in this file is irrelevant to the content and is needed only to generate the TOC. I think it's a little bit too much =)

I don't understand where you're getting those numbers. Here's the basic example from the new readme:

* Heading
:PROPERTIES:
:TOC:      :include all
:END:

This text appears before the TOC.

:CONTENTS:
- [[#heading][Heading]]
  - [[#subheading][Subheading]]
:END:

This text appears after it.

** Subheading

There are only 5 lines related to configuring this example TOC, which is the simplest one possible:

:PROPERTIES:
:TOC:      :include all
:END:
:CONTENTS:
:END:

You included the before-save-hook line in your comparison, but that's optional and a matter of user preference.

I prefer to set it as a buffer-local variable in documents where I want a TOC to be generated automatically, because I don't use TOCs in all of my Org documents, and I wouldn't want to run the TOC-generating function when I save most of my Org files.

In the toc-org readme you suggest adding the TOC-generating function to the user's global Org configuration, which you could also do with org-make-toc. And users of toc-org could also use buffer-local variables instead of adding it to the global Org configuration. So that's not a difference between the packages.

Maybe, if you think about TOCs everyday and find it fascinating then it's ok, but for a regular person, I think, 1 functional line is more than enough =)

I developed org-make-toc so I could generate flexible TOCs for complex documents. For example, at https://github.com/alphapapa/emacs-package-dev-handbook, using this new 0.5 branch, I have top-level TOCs as well as subtree-local TOCs: the document begins with the top-level Contents TOC, with links to only the top-level sections. Next comes the Emacs Lisp section, which includes a subtree-local TOC showing headings 1 level deep. The next section is Animations / Screencasts which includes its own subtree-local TOC, this time showing headings 2 levels deep. Then there's this section, which has its subtree's TOC inline, after an introductory paragraph.

These flexible TOCs are important to enable easy navigation through the long document, and to give users an overview of each section's content when they arrive at one. As well, they can be very helpful when producing a complex document like a book or dissertation, as shown by the examples.org file.

So, my switch is not today - again =)

That's okay, you don't need to switch! :) Ideally, I'd like to improve org-make-toc to meet more users' needs in a flexible, simple way, including yours. But your solution meets your needs, and that's what matters. I appreciate your feedback very much as a fellow Emacs user and developer of a similar package.

[snosov1]

I just tried that and it works. Did you accidentally use the README.org from master instead of this wip/rewrite branch?

No. I use wip/rewrite. Maybe it has something to do with my Org version (8.2.10). The call to (looking-at-p org-drawer-regexp) returns nil because the value of org-drawer-regexp equals to "^[ ]*:PROPERTIES:[ ]*$"

I think the only way to solve that

In toc-org I solve it via org-link-translation-function.

Ideally GitHub's renderer would be updated to handle Org-native links better

A man can dream... =)

However, AFAICT, toc-org has the same problem.

It should work. Do you have toc-org-mode enabled?

But that seems to me to be orthogonal to the function of creating TOCs.

I see your point, but can't say I fully agree. Following the same logic, collecting a TOC and generating hrefs for various backends could be 2 (or more) different packages. Where's the end of this?

I feel like that's pretty straightforward. As Akira said, it would be helpful to have a command to help with configuring TOC properties, so I'll probably add one.

Well, obviously, I can't say that following a simple 4 step instruction is not straightforward. But the mere fact that such questions/comments even come up and that you need to add new functions to help to use old functions also tells something.

which has significant benefits

I understand that these features you mention is the main value proposition/benefit of your package. It's just, as I mentioned - "your" features is priority number 5 in my book. If I don't get priorities 1-4, I don't really care about adding text before or after the TOC or about precise control about inclusion/exclusion.

I guess you mean that the TOC-making command should be idempotent. I agree, and AFAIK, this is the case with org-make-toc, both in 0.5-pre and earlier versions. Am I misunderstanding you?

Probably, it works ok, but I couldn't test it.

I'm not sure what you mean. Would you elaborate?

There's certain logic with how GitHub (following Org) treats certain states (TODO, DONE, COMMENT, custom-defined states). toc-org acknowledges that so the generated hrefs work when rendered by GitHub.

that can't be fixed without using ID or CUSTOM_ID properties,

In toc-org I implement it via hash table.

I confess that I have no interest in Markdown, given that we have Org. :)

Well, me, too. But non-Emacs people use Markdown predominantly. Adding it to toc-org's implementation was fairly straightforward with all the other features (including C-c C-o jumps). So, why not to have it as priority number 5 (after 1-4 are implemented)? =)

I don't understand where you're getting those numbers.

I took your README.org and measured, that's it. Not the basic example, your entire readme. In the basic example the ratio is even worse. You have 5 "functional lines":

:PROPERTIES:
:TOC:      :include all
:END:
:CONTENTS:
:END:

And only 4 content lines (6 if count the generated TOC entries).

* Heading
This text appears before the TOC.
This text appears after it.
** Subheading

In other words, there's more lines related to TOC generation compared to the actual content =)

You included the before-save-hook line in your comparison,

I didn't include it for the reasons you mention. But you did include it in your readme even though it's not the best option =)

I appreciate your feedback very much as a fellow Emacs user and developer of a similar package.

Cheers! =)

[easbarba]

Sorry for the late response, done in https://github.com/lxbarbosa/novels.org/issues/6

Thanks :)