Updated 2context vim to process line macros

[hernot]

Hi as proposed on feature request #29 after some testing i hereby open the pullrequest formy suggestions on how to improve use of linemacros (\someline, \startline and \stopline).

line macros are still to be used inside comments only
escape parameter may be set to true be needent
showlinetags parameter (currently only in vim script 2context.vim) for additionally typesetting macros primarily for document debugging purpose
comments which only contain line macros are hidden
    line is typeset as emtpy if not containing anything than comment
    inlinecomment is simply replaced by empty string (could be changed to single whitespace)

The only file changed so far for starting discussion is 2context.vim all other files stay unchanged. Currently default value for showlinetags is 1 for debugging purpose.

NOTE: fixes and replaces pullrequest #30

TODO: -make showlinetags parameter available within context -discuss possible improvements of suggested functionality

[hernot]

Ok to answer the remainder of your comment on the now closed pullrequest #30

• Add a paragraph in vim-README.md explaining the usage

I'll do as soon as i have successfully pulled showlinemacros parameter through to context surface

• Add a few test files in tests/vim/3{1,2,..}-*.tex with basic usage. I manually compile all these files to see that new changes don't break any functionality. Such tests will ensure that the line numbering is always tested.

I'll do as soon as i have successfully pulled showlinemacros parameter through to context surface

If we are going this route (scan code for strings and reinsert then after the conversion by 2context), it might make sense to follow the convention used in the built-in typing environments and scan for /BTEX ... /ETEX occuring anywhere in the code (even outside comments) and reinsert the content at the beginning (or end?) of the current line after conversion. The logic will not be too different from what you are currently doing, but this is a general interface which will work for commands other than line numbering (e.g., callouts, like in docbook, or some fancy metapost stuff).

I descided not to do so for the following two reasons

1. i tried to keep things simple and with the chosen approach only comment strings must be scanned for \someline, \startline and \stopline.
2. Outside comments the line must be scanned for the tags and the tags must be removed before line is passed to syntaxhighlighting tokenizer. -> a lot more work, more ugly code for just a handful of tagged lines a huge impact on performance.
3. huge effort to change script including introduciton of functions etc. wanted to keep changes simple

So if you aggee i would not opt for scanning outsied comments and if currently only the linemacros should be handled specially independent of actual setting of escape i would keep it for now as is unless you plan to enforce usage of /BTEX /ETEX for all macros used inside comments

[hernot]

need some help/advise from your side: tried to put showlinetagsparameter from vimscript through to context. i looked as the escape parameter as an example but i just get an Undefined controlsequence when i just copy the \setvalue lines for escape and replace the parameter name.

Something I'm missign or not getting how to tell tex that the parameter/attribute is valid. Where do i find the appropriate doc how to add that new parameter. I seem to not get it.

[adityam]

Something I'm missign or not getting how to tell tex that the parameter/attribute is valid. Where do i find the appropriate doc how to add that new parameter. I seem to not get it.

That should work, but it is difficult to say what went wrong without looking at the exact change that you made in t-vim.tex. There is a limit on how many -c... flags can be passed and it is possible that we are hitting that limit

[adityam]

If we are going this route (scan code for strings and reinsert then after the conversion by 2context), it might make sense to follow the convention used in the built-in typing environments and scan for /BTEX ... /ETEX occuring anywhere in the code (even outside comments) and reinsert the content at the beginning (or end?) of the current line after conversion. The logic will not be too different from what you are currently doing, but this is a general interface which will work for commands other than line numbering (e.g., callouts, like in docbook, or some fancy metapost stuff).

I descided not to do so for the following two reasons

1. i tried to keep things simple and with the chosen approach only comment strings must be scanned for \someline, \startline and \stopline.
2. Outside comments the line must be scanned for the tags and the tags must be removed before line is passed to syntaxhighlighting tokenizer. -> a lot more work, more ugly code for just a handful of tagged lines a huge impact on performance.
3. huge effort to change script including introduciton of functions etc. wanted to keep changes simple

So if you aggee i would not opt for scanning outsied comments and if currently only the linemacros should be handled specially independent of actual setting of escape i would keep it for now as is unless you plan to enforce usage of /BTEX /ETEX for all macros used inside comments

Any particular reason you want \startline etc to work even when escape is disabled. Is there some disadvantage of enabling escape in your setup?

I prefer general interfaces compared to specific interface for each macro. I also created an implementation for /BTEX .. /ETEX in #33. This works outside the comment environment as well (see the test file in tests/vim/41-btex.tex for an example).

[hernot]

Any particular reason you want \startline etc to work even when escape is disabled. Is there some disadvantage of enabling escape in your setup?

Nope there isn't any issue or reason to me. Just that Line numbering macros work in other typing environments without the need to set any escape parameter and why should that than be necessary in vim Typing? That was all.

[hernot]

Something I'm missign or not getting how to tell tex that the parameter/attribute is valid. Where do i find the appropriate doc how to add that new parameter. I seem to not get it.

That should work, but it is difficult to say what went wrong without looking at the exact change that you made in t-vim.tex. There is a limit on how many -c... flags can be passed and it is possible that we are hitting that limit

It so far has nothing todo with the setup of the actual filter command, there there i followed your comment providing the following advise % vim only accepts 10 -c commands, so we combine a few let statements

I next took the lines for escape, duplicated them and replaced escape by showlinetags the so that they read as follows

\setvalue{\vimtyping@id-\c!showlinetags-\v!off}{0}
\setvalue{\vimtyping@id-\c!showlinetags-\v!on}{1}

see also attached file. With this (ok i ju Hm, is it sufficient to just put the local copy onto the TEXINPUTS path to use the modified version instead of the one installed from the t-vim package. Currently i just moved the original files aside and linked the ones modified on my local working directory into the package directory. This worked for 2context.vim so i thought that might work also the tex file, but could that be the reason. How would i have to setup my local working copy to be testable the canonical way?

[hernot]

I prefer general interfaces compared to specific interface for each macro. I also created an implementation for /BTEX .. /ETEX in #33. This works outside the comment environment as well (see the test file in tests/vim/41-btex.tex for an example).

I would also be OK looking for /BTEX .. /ETEX pair enclosing the linetag macros that is just a small change to the regular expression. But i do not see any benefit in allowing these Tags outside comments. Examples 2 and 3 are to me more readable and understandable from context document source only than example 1. In other words i do prefere that code actualy typed insided the context source file is as close to what is typeset and what is compilable or interpretable by the language compiler or interpreter. And comments are usually those elements by which compilers and interpreters allow to place arbitrary text and stuff.

\startPython
def example1(a,b): 
    return a+b /BTEX \someline[code:example1:sumline] /ETEX
\stopPython

\startPython
def example2(a,b): 
    return a+b # /BTEX \someline[code:example1:sumline] /ETEX
\stopPython

\startPython
def example3(a,b): 
    return a+b # \someline[code:example1:sumline]
\stopPython

In case it is really necessary that vim typing supports all fileformats suppored by vim syntaxhighlightin, including those confirguration formats which do not forsee any documentation format i would add an additoinal switch eg. linetags=[anywhere,comments] allowing users to decide themselves whether they allways want to place them inside comments only or have to place them on any line eg. cause syntax does not forsee any comment or alike general descriptive text. But than i would suggest to append /BTEX /ETEX pair to the list of comment symbols recognized by the language. That would allow to denote them ad comment symbols in case the selected syntax does not know the concept of comments at all.

[adityam]

Hm, is it sufficient to just put the local copy onto the TEXINPUTS path to use the modified version instead of the one installed from the t-vim package. Currently i just moved the original files aside and linked the ones modified on my local working directory into the package directory. This worked for 2context.vim so i thought that might work also the tex file, but could that be the reason. How would i have to setup my local working copy to be testable the canonical way?

I do the following:

1. Clone the git repo

2. Make changes in t-vim and 2context.vim according to my git workflow.

3. Create appropriate test files in tests/vim/...

4. Run context --mode=dev-vim tests/vim/.... The dev-vim mode makes sure that the local copy of the module is used.

So, there is no need to mess around with files in the context tree.

[adityam]

On Sat, 14 Mar 2020, hernot wrote:

Any particular reason you want \startline etc to work even when escape is disabled. Is there some disadvantage of enabling escape in your setup?

Nope there isn't any issue or reason to me. Just that Line numbering macros work in other typing environments without the need to set any escape parameter and why should that than be necessary in vim Typing? That was all.

Actually line numbering does not work in other typing environments unless you put them in a /BTEX ... /ETEX tags. Originally escape=on was copying a similar feature in one of the latex modules (which allows you to get math equations typeset in the comments).

[adityam]

On Sat, 14 Mar 2020, hernot wrote:

I prefer general interfaces compared to specific interface for each macro. I also created an implementation for /BTEX .. /ETEX in #33. This works outside the comment environment as well (see the test file in tests/vim/41-btex.tex for an example).

I would also be OK looking for /BTEX .. /ETEX pair enclosing the linetag macros that is just a small change to the regular expression. But i do not see any benefit in allowing these Tags outside comments.

That is how they are implementing the default typing environment as well as in m-scite (which uses scite for syntax highlighting). These environments all the "escape code" /BTEX .. /ETEX to be configurable. See this MyWay for details:

http://pragma-ade.nl/general/magazines/mag-1102-mkiv.pdf

Examples 2 and 3 are to me more readable and understandable from context document source only than example 1, as i do prefere that code inside document source is as easy to read and understand like it is on the typeset document.

Further if the code is included from an external source file than this is not an option anyway as it is likely that the main purpose of this file is to be read and processed by the language compiler or interpreter.

I agree with these points. I have two concerns:

1. I don't want to specialize the logic to linenumering. If we are adding this feature, then we might as well allow abitrary ConTeXt commands. For example, then I can create a anchored node and draw an arrow to a specific line and so on.

2. For general code, how do we distinguish between "<comment marker>" (where the comment marker should be removed), and "" (inserted by the user, where the comment marker should not be removed).

Your pull request takes care of 2 but not 1. My pull request takes care of 1 but not 2. I am not sure what is the best way to achieve both these features.

[hernot]

Hi I think i have found a really cool way to make /BTEX /ETEX fully work thoughout code independent from comments. See my updated commit for a first set of examples. See also the commentes especially inside vim/syntax/vimtypingfilter.vim special syntax file. Some languages and language families need some special tuning or switchcase setup inside that file. I just for now, for demonstration purpose die a rather generic syntaxgroup for now.

The cool stuff is all done by vim syntax through synID, synIDtrans and synIDattr no tinkering with regular expressions, hiding comments etc. See my two examples 33 and 34

Looking forward to your review and improvement suggestions.

[adityam]

Hi I think i have found a really cool way to make /BTEX /ETEX fully work thoughout code independent from comments. See my updated commit for a first set of examples. See also the commentes especially inside vim/syntax/vimtypingfilter.vim special syntax file.

Thank you for the updated code. This is definitely a cleaner solution the both our previous regex based solutions.

Some languages and language families need some special tuning or switchcase setup inside that file. I just for now, for demonstration purpose die a rather generic syntaxgroup for now.

This can also be done as part of 2context.vim without requiring an additional syntax file. I'll play around with this code to make sure that there are no corner cases and then merge the code.

This may take a few weeks because of everything else that is happening.

[adityam]

Could you squash both these changes in your fork and update the pull request. Otherwise the diff in the second commit are with respect to the original pull request.

[hernot]

OK done Beeing courious how you would do it without syntax file, form vim docs, stackoverlfow and other references i had the impression that syntax can only be extended through custom syntax file.

[adityam]

Beeing courious how you would do it without syntax file, form vim docs, stackoverlfow and other references i had the impression that syntax can only be extended through custom syntax file.

(Untested):

Simply adding the line

syntax region ....

in 2context.vim at appropriate place will also work. My understanding is that set syntax=filetype simply sources filetype.vim file from $VIMRUNTIME/syntax/filetype.vim. So, just executing those lines directly also works.

[hernot]

Would be cool if that works. Besides this one thing has to be taken care of. While browsing part of the syntaxfiles distributed along with vim i figured the following ( i have put it anyway in the vimtypingfilter.vim file too) is that different families of languages use quite diverse namings for concepts known form c/c++/c#/java or python as code Block, Region etc. Therefor the syntax group name pattern i have used in my proof of concept implementation is a rather wild and coarse guess. It is mostly based upon my knowledge of the aforementioned languages. In other words i do fear that for other languages better language families it will be necessary to manage the appropriate definition vimTypingTexTags group in a sort of switch case or if then elif then else structure. For example in one language Region or Block means foldable Code segment and codeblock enclosed in {} while in another it can mean some compact sturcture which should not, to stay readable and understandable, be littered by /BTEX /ETEX sequences.

Therefor i do suggest the following approach. 1) start with the languages we two use regularly (mine i already have named) and tune the configuration for group name pattern towards them 2) if feature requests are filed for languages which are not properly supported by the initial or default setup extend the corresponding section in 2context.vim accordingly.

[adityam]

I have made some changes to simplify the code in 2context.vim. See updated branch. The following needs to be done, before this can be merged.

• [x] Should this feature be enabled by default, or should we add an option, say texcomment=(yes|no) to enable it. I am in favor of adding such an option which is disabled by default, but I am interested in hearing your opinion on this as well. If we decide to add such an option, then I can work on it because I know what needs to be done.

• [x] We need to document the feature (and the option, if we decide to add it), in vim-README.md. Could you please write that.

• [x] [This is more of a reminder for me]. Updated the dates in 2context.vim and t-vim.tex. I always do this as the last step before a new release.

• [x] Run the test suite to make sure that there is no side effect on any other feature. (Unlikely, but I always check before a new release). I will do this.

Once again, thanks a lot for persisting on this. I always wanted to add this but was afraid that it will lead to complicated (and therefore difficult to maintain code). But your solution is very clean, and in the end requires very little changes in the file.

[hernot]

I have made some changes to simplify the code in 2context.vim. See updated branch. The following needs to be done, before this can be merged.

* [ ]  Should this feature be enabled by default, or should we add an option, say `texcomment=(yes|no)` to enable it. I am in favor of adding such an option which is disabled by default, but I am interested in hearing your opinion on this as well. If we decide to add such an option, then I can work on it because I know what needs to be done.

Yes it should. Afaik that would be consistent with verbatim environments. And having to activate this explicitly reduces the chance that users litter their sources allover with /BTEX /ETEX sections allover without thinking afaik for Verbatim and Typing environments this is what tthe escape or so is for.

* [ ]  We need to document the feature (and the option, if we decide to add it), in `vim-README.md`. Could you please write that.

Yes as soon as the final name is known i will update README-vim.md. If you are ok i would also add a small hint that it is not wise to include /BTEX /ETEX sections within strings, and other code elements which readers might copy and paste into a file, they fail to successfully compile cause of non literal stuff.

* [ ]  [This is more of a reminder for me]. Updated the dates in `2context.vim` and `t-vim.tex`. I always do this as the last step before a new release.

* [ ]  Run the test suite to make sure that there is no side effect on any other feature. (Unlikely, but I always check before a new release). I will do this.

Once again, thanks a lot for persisting on this. I always wanted to add this but was afraid that it will lead to complicated (and therefore difficult to maintain code). But your solution is very clean, and in the end requires very little changes in the file.

Wellcome.

[adityam]

I have made some changes to simplify the code in 2context.vim. See updated branch. The following needs to be done, before this can be merged.

* [ ]  Should this feature be enabled by default, or should we add an option, say `texcomment=(yes|no)` to enable it. I am in favor of adding such an option which is disabled by default, but I am interested in hearing your opinion on this as well. If we decide to add such an option, then I can work on it because I know what needs to be done.

Yes it should. Afaik that would be consistent with verbatim environments. And having to activate this explicitly reduces the chance that users litter their sources allover with /BTEX /ETEX sections allover without thinking afaik for Verbatim and Typing environments this is what tthe escape or so is for.

Unfortunately, I have already used escape=on to indicate using TeX commands in comments. One option is to use the following: escape can take a list, the value comment will enable the previous behavior of escape=on and the value command will enable the /BTEX and /ETEX escape. For backward compatibility, escape=on is equal to escape=comment. One can enable both behaviors using escape={comment,command}.

* [ ]  We need to document the feature (and the option, if we decide to add it), in `vim-README.md`. Could you please write that.

Yes as soon as the final name is known i will update README-vim.md. If you are ok i would also add a small hint that it is not wise to include /BTEX /ETEX sections within strings, and other code elements which readers might copy and paste into a file, they fail to successfully compile cause of non literal stuff.

Yes, that is okay.

[hernot]

Unfortunately, I have already used escape=on to indicate using TeX commands in comments. One option is to use the following: escape can take a list, the value comment will enable the previous behavior of escape=on and the value command will enable the /BTEX and /ETEX escape. For backward compatibility, escape=on is equal to escape=comment. One can enable both behaviors using escape={comment,command}.

HM I would not explicitly allow mixing both behaviors within the same start<t-vim> stop<t-vim> text segment. I would limit mixed behaviour to parameter on for some limited time printing a warning message in case escape is on without any /BTEX /ETEX section but comments that on will be switched to command value only in future as comment is deprecated and only supported for sake of old ConTeXt sources still working for some time. or at least put that warning in the README.vim.md

[adityam]

Unfortunately, I have already used escape=on to indicate using TeX commands in comments. One option is to use the following: escape can take a list, the value comment will enable the previous behavior of escape=on and the value command will enable the /BTEX and /ETEX escape. For backward compatibility, escape=on is equal to escape=comment. One can enable both behaviors using escape={comment,command}.

HM I would not explicitly allow mixing both behaviors within the same start<t-vim> stop<t-vim> text segment. I would limit mixed behaviour to parameter on for some limited time printing a warning message in case escape is on without any /BTEX /ETEX section but comments that on will be switched to command value only in future as comment is deprecated and only supported for sake of old ConTeXt sources still working for some time. or at least put that warning in the README.vim.md

I don't see a reason to break backward compatibility. There are definitely uses who use the current escape=on (it was implemented based on a request to imitate similar behavior of listing package in LaTeX). There are pros and cons of both approaches and that is the reason I want to have keep both behaviors.

[hernot]

I don't see a reason to break backward compatibility. There are definitely use s who use the current escape=on (it was implemented based on a request to imitate similar behavior of listing package in LaTeX). There are pros and cons of both approaches and that is the reason I want to have keep both behaviors.

Ah OK. I have never used the listing package in LaTeX eventhough all my relevant documents are in LaTeX.

But than i would call the values as follows on ... for now mixed mode lateron only command latex or listing ... for LaTeX - listing compatibility mode command or verbatim or typing ... for ConTeXt typing and verbatime like behavior off ... no escape at all

And how about providing t-vim users with an vimtypingsetupdefault command which takes for now just an escape = latex and escape=command parameter by which users can select what the default behavior throughout their whole document for escape=on should be. The default setting would be for now latex lateron if it turns out most t-vim users set it to command anyway that could be swithched and LaTeX-listing style users would than need only to insert a single line into their preamble documents to get their desired behavior back.

Mixing within a single code block (\start<vimtyping> \stop<vimtyping>) i do not like.

[adityam]

I prefer comment (enabled only in comments) and command (enabled using /BTEX ... /ETEX commands). Easy to remember.

I understand that no one will probably use both at the same time but since both can be selected at the vim end, why disable selecting both of them at the tex end?

[hernot]

I just would not tell them on tex end that it would be possible to activate both at the same time. If you tell them they will use it that mess and complain that that mess causes problems non of us two desires and wants to support. If you dont tell them than less work less nasty support requests and issue reports from people who use that mixing feature just cause it is stated that it is possible. Users should either use comment mode or command mode they can anyway switch on every \start<vimtyping> \stop<vimtyping> block. That should be sufficient.

And whether on is equal to comment to keep for now old document sources working and will later switch to command can be decided any time later.

[adityam]

I just would not tell them on tex end that it would be possible to activate both at the same time. If you tell them they will use it that mess and complain that that mess causes problems non of us tẃo desires and wants to support.

There is no interaction between the two mechanisms, so I don't see why there will be an issue. I added a simple test on my branch on github, and that works fine.

And whether on is equal to comment to keep for now old document sources working and will later switch to command can be decided any time later.

I don't want to break backward compatibility. I have code written 10 years ago using this module that I still use.

[hernot]

I think we are thinking about two different things.

Ensuring that existing document sources i do fully agree with you. Providing users who switch from LaTeX-listing to ConTeXt-T-vim or simply prefer the LaTeX-listing approach over the ConTeXt approach escaped-comment style is a reasonable and thus good idea to me too. Thus my remaining point is about that escape={comment,command} does not make any reasonable sense to me. Likely that i missed somewhen throughout the discussion that there is already a final descision on this minor side point or that it does not need any as not relevant. If so i apologizes for still chewing on it.

[adityam]

I think we are thinking about two different things.

Agreed. I updated the branch with the following features:

Add options to enable escape mechanism at tex end

• escape=on or escape=comment enables escape in comments
• escape=command enables escape in /BTEX .. /ETEX (anywhere)
• Both cannot be enabled simultanenously.

I also updated the test files accordingly.

[adityam]

I also disabled both features by default in \inline.... I don't see any reason why anyone will use this feature in an inline code snippet.

[adityam]

I checked all the tests and added documentation. Take a look at the updated branch. I will do some more tests, and release a new version tomorrow.

[adityam]

Merged!