Use org-mode for docs instead of rst-mode

[zedeus]

Using a bunch of regex, the raw rst-docs can be converted into org-mode formatting. This enables some neat features, like using org-babel for inline Nim syntax highlighting, section folding, and file links.

It would be nice if the inline documentation links, such as `tables module <tables.html>`_ and `resume proc <#resume,Process>`_ linked to the respective documentation, but I think that would require some custom org link function calling nimsuggest-show-doc and finding the linked symbol, I have no idea how to achieve this.

Included is also a fix for evil-mode users, which makes nimsuggest-doc-mode-map take precedence over global evil bindings (copied from syl20bnr/spacemacs#10226)

Here are two examples showing before and after (click to view full-res):

Here the OrderedTable example is collapsed:

[kaushalmodi]

I'm curious.. would just using pandoc be faster to do this conversation? May be use pandoc if it is available in PATH? Else use the replace-regexp approach?

I might be wrong, but using just regexp, we might not be able to convert from rst tables to Org tables. But Pandoc can convert those too.

Related: I do the opposite here; I convert Org documentation for a Nim package to RST so that nim doc can understand using Pandoc (look at the docs task in strfmt.nimble).

[zedeus]

I suppose using pandoc may be possible, though from my testing regexp seemed fast enough to not be noticable. Converting tables isn't really possible with regexp indeed, but imo they look fine as is so I don't mind.

[kaushalmodi]

Actually, my concern was not that much with speed than with the accuracy and coverage of conversation from RST to Org.

In any case, I am looking forward to the progress of this PR. Thanks for working on this.

[zedeus]

Ah, I tested with a bunch of functions and modules to make sure formatting was consistently correct. Feel free to test it out and see if you can find any mistakes.

[zedeus]

Here's a comparison of rst, my regexp, and pandoc (using the online demo). I'm not sure if pandoc has a bunch of settings to modify the output, but in this case it doesn't work very well.

Raw RST

```rst sequtils Signature ========= skModule Documentation ============= Although this module has ``seq`` in its name, it implements operations not only for `seq`:idx: type, but for three built-in container types under the ``openArray`` umbrella: * sequences * strings * array The system module defines several common functions, such as: * ``newseq[T]`` for creating new sequences of type ``T`` * ``@`` for converting arrays and strings to sequences * ``add`` for adding new elements to strings and sequences * ``&`` for string and seq concatenation * ``in`` (alias for ``contains``) and ``notin`` for checking if an item is in a container This module builds upon that, providing additional functionality in form of procs, iterators and templates inspired by functional programming languages. For functional style programming you have different options at your disposal: * pass `anonymous proc`_ * import `sugar module`_ and use `=> macro.m,untyped,untyped>`_ * use `...It templates<#18>`_ (`mapIt<#mapIt.t,typed,untyped>`_, `filterIt<#filterIt.t,untyped,untyped>`_, etc.) The chaining of functions is possible thanks to the `method call syntax`_. .. code-block:: import sequtils, sugar # Creating a sequence from 1 to 10, multiplying each member by 2, # keeping only the members which are not divisible by 6. let foo = toSeq(1..10).map(x => x*2).filter(x => x mod 6 != 0) bar = toSeq(1..10).mapIt(it*2).filterIt(it mod 6 != 0) doAssert foo == bar echo foo # @[2, 4, 8, 10, 14, 16, 20] echo foo.any(x => x > 17) # true echo bar.allIt(it < 20) # false echo foo.foldl(a + b) # 74; sum of all members .. code-block:: import sequtils from strutils import join let vowels = @"aeiou" # creates a sequence @['a', 'e', 'i', 'o', 'u'] foo = "sequtils is an awesome module" echo foo.filterIt(it notin vowels).join # "sqtls s n wsm mdl" ---- **See also**: * `strutils module`_ for common string functions * `sugar module`_ for syntactic sugar macros * `algorithm module`_ for common generic algorithms * `json module`_ for a structure which allows heterogeneous members Location ======== /home/zed/.choosenim/toolchains/nim-#devel/lib/pure/collections/sequtils.nim ```

Regexp Org

```org sequtils * Signature skModule * Documentation Although this module has ~seq~ in its name, it implements operations not only for seq type, but for three built-in container types under the ~openArray~ umbrella: - sequences - strings - array The system module defines several common functions, such as: - ~newseq[T]~ for creating new sequences of type ~T~ - ~@~ for converting arrays and strings to sequences - ~add~ for adding new elements to strings and sequences - ~&~ for string and seq concatenation - ~in~ (alias for ~contains~) and ~notin~ for checking if an item is in a container This module builds upon that, providing additional functionality in form of procs, iterators and templates inspired by functional programming languages. For functional style programming you have different options at your disposal: - pass ~anonymous proc~ - import ~sugar module~ and use ~=> macro~ - use ~...It templates~ (~mapIt~, ~filterIt~, etc.) The chaining of functions is possible thanks to the ~method call syntax~. #+BEGIN_SRC nim import sequtils, sugar # Creating a sequence from 1 to 10, multiplying each member by 2, # keeping only the members which are not divisible by 6. let foo = toSeq(1..10).map(x => x*2).filter(x => x mod 6 != 0) bar = toSeq(1..10).mapIt(it*2).filterIt(it mod 6 != 0) doAssert foo == bar echo foo # @[2, 4, 8, 10, 14, 16, 20] echo foo.any(x => x > 17) # true echo bar.allIt(it < 20) # false echo foo.foldl(a + b) # 74; sum of all members #+END_SRC #+BEGIN_SRC nim import sequtils from strutils import join let vowels = @"aeiou" # creates a sequence @['a', 'e', 'i', 'o', 'u'] foo = "sequtils is an awesome module" echo foo.filterIt(it notin vowels).join # "sqtls s n wsm mdl" #+END_SRC ---- *See also*: - ~strutils module~ for common string functions - ~sugar module~ for syntactic sugar macros - ~algorithm module~ for common generic algorithms - ~json module~ for a structure which allows heterogeneous members * Location [[file:/home/zed/.choosenim/toolchains/nim-#devel/lib/pure/collections/sequtils.nim][/home/zed/.choosenim/toolchains/nim-#devel/lib/pure/collections/sequtils.nim]] ```

Pandoc Org

```org sequtils * Signature :PROPERTIES: :CUSTOM_ID: signature :END: skModule * Documentation :PROPERTIES: :CUSTOM_ID: documentation :END: Although this module has =seq= in its name, it implements operations not only for =seq= type, but for three built-in container types under the =openArray= umbrella: * sequences * strings * array The system module defines several common functions, such as: * =newseq[T]= for creating new sequences of type =T= * =@= for converting arrays and strings to sequences * =add= for adding new elements to strings and sequences * =&= for string and seq concatenation * =in= (alias for =contains=) and =notin= for checking if an item is in a container This module builds upon that, providing additional functionality in form of procs, iterators and templates inspired by functional programming languages. For functional style programming you have different options at your disposal: * pass [[file:manual.html#procedures-anonymous-procs][anonymous proc]] * import [[file:sugar.html][sugar module]] and use [[][=> macro.m,untyped,untyped>]] * use [[#18][...It templates]] ([[#mapIt.t,typed,untyped][mapIt]], [[#filterIt.t,untyped,untyped][filterIt]], etc.) The chaining of functions is possible thanks to the [[file:manual.html#procs-method-call-syntax][method call syntax]]. #+BEGIN_EXAMPLE # Creating a sequence from 1 to 10, multiplying each member by 2, # keeping only the members which are not divisible by 6. let foo = toSeq(1..10).map(x => x*2).filter(x => x mod 6 != 0) bar = toSeq(1..10).mapIt(it*2).filterIt(it mod 6 != 0) doAssert foo == bar echo foo # @[2, 4, 8, 10, 14, 16, 20] echo foo.any(x => x > 17) # true echo bar.allIt(it < 20) # false echo foo.foldl(a + b) # 74; sum of all members #+END_EXAMPLE #+BEGIN_EXAMPLE let vowels = @"aeiou" # creates a sequence @['a', 'e', 'i', 'o', 'u'] foo = "sequtils is an awesome module" echo foo.filterIt(it notin vowels).join # "sqtls s n wsm mdl" #+END_EXAMPLE -------------- *See also*: * [[file:strutils.html][strutils module]] for common string functions * [[file:sugar.html][sugar module]] for syntactic sugar macros * [[file:algorithm.html][algorithm module]] for common generic algorithms * [[file:json.html][json module]] for a structure which allows heterogeneous members * Location :PROPERTIES: :CUSTOM_ID: location :END: /home/zed/.choosenim/toolchains/nim-#devel/lib/pure/collections/sequtils.nim ```

[krux02]

Well, this is definitively cool. I also really like the result. But I don't think I can merge it right now. The reason for it is simple, the documentation of Nim isn't written in org, it is written in rst. When it is visualized in org-mode it is technically a lie to the user. The second reason is, it might fail in some cases. If you would implement it as an option that can be enabled via customize and describe it as experimental, then I wouldn't see any reason anymore why this shouldn't be merged.

[zedeus]

@krux02 I've made it optional now, off by default. I'm not sure if this is the best approach but it seems to be working fine. Also made it hide the documentation section if it's empty. I suppose adding the setting to the readme would be a good idea, as most users would probably prefer the org-mode formatting, but I'm not sure where to put it.

[krux02]

last bit, can you split nimsuggest--show-doc in nimsuggest--show-doc-rst and nimsuggest--show-doc-org, where nimsuggest--show-doc-rst is the unchanged old nimsuggest--show-doc function? For customization I think it would be the best to have a customizable nimsuggest--show-doc-function symbol.

[zedeus]

@krux02 Implementing that lead to somewhat simpler code, however I can't figure out how to set the default of the custom variable to nimsuggest--show-doc-rst. The defcustom calls are in nim-vars.el, which does not and cannot require nim-mode, since that would lead to a recursive import, so the functions from nim-suggest.el are undefined. I've pushed a solution using 'rst and 'org symbols, let me know what you think.

[krux02]

@zestyr Sorry for the late reply, I replied but somehow the message is missing here. I must have forgotten to actually press comment. Anyway ...

The code simplification was intentional ;)

If I understand elisp correctly, then you don't need to know the symbol nimsuggest--show-doc-rst to use it. You should be able to just set nimsuggest--show-doc-function to 'nimsuggest--show-doc-rst only when you later try to call nimsuggest--show-doc-rst via (funcall nimsuggest--show-doc-function) the function needs to be defined by then.

This is how I imagined it.

(defcustom nimsuggest--show-doc-function 'nimsuggest--show-doc-rst
  "The style used to display Nim documentation.
Options available are `nimsuggest--show-doc-rst' and
`nimsuggest--show-doc-org'.  Note `nimsuggest--show-doc-org'
enables syntax highlighting and section folding, but it is
experimental."
  :type 'function
  :group 'nim)

;; the when fboundp part is probably not necessary when you put it in a file where also nimsuggest--show-doc-rst is defined
(when (fboundp nimsuggest--show-doc-function) 
  (funcall nimsuggest--show-doc-function))

[zedeus]

Indeed that works, thank you. I picked nimsuggest-show-doc-function as -- is a convention indicating it's an internal symbol.

[zedeus]

@krux02 fixed the typo, thank you.