Implement help system

[eteran]

I think I may choose to "port" many of the help pages to the GitHub Wiki as opposed to being actually built into nedit-ng itself. This would be more dynamic and potentially community-driven.

But we should have some sort of help when the user clicks on... help :-)

Mentioned by @anjohnson in issue #120

[eteran]

@anjohnson, I have started some initial efforts to get the still relevant contents of the help system into the nedit-ng wiki!

I hope this helps.

[anjohnson]

Thanks. I assume your text for the initial Wiki pages was sourced from the nedit-ng/doc/src tree. I think pandoc would make quick work of the html to gfm (GitHub Flavored Markdown) conversions, and some regex edits could add some of the missing formatting. I'd be happy to add some of the other pages, but I can't promise to spot all of the places that will need manually fixing up due to the new implementation.

Here's a verbatim pandoc conversion of 16.html as an example (of a page which you've already posted. It has some missing formatting and differences in the generated gfm output that I would fix (and it looks like the gfm in an issue comment isn't line-wrapped quite like a Wiki page).

---

Regular expressions (regex's) are useful as a way to match inexact sequences of characters. They can be used in the 'Find...' and 'Replace...' search dialogs and are at the core of Color Syntax Highlighting patterns. To specify a regular expression in a search dialog, simply click on the 'Regular Expression' radio button in the dialog.

A regex is a specification of a pattern to be matched in the searched text. This pattern consists of a sequence of tokens, each being able to match a single character or a sequence of characters in the text, or assert that a specific position within the text has been reached (the latter is called an anchor.) Tokens (also called atoms) can be modified by adding one of a number of special quantifier tokens immediately after the token. A quantifier token specifies how many times the previous token must be matched (see below.)

Tokens can be grouped together using one of a number of grouping constructs, the most common being plain parentheses. Tokens that are grouped in this way are also collectively considered to be a regex atom, since this new larger atom may also be modified by a quantifier.

A regex can also be organized into a list of alternatives by separating each alternative with pipe characters, '|'. This is called alternation. A match will be attempted for each alternative listed, in the order specified, until a match results or the list of alternatives is exhausted (see Alternation section below.)

The 'Any' Character

If a dot ('.') appears in a regex, it means to match any character exactly once. By default, dot will not match a newline character, but this behavior can be changed (see help topic Parenthetical Constructs, under the heading, Matching Newlines).

Character Classes

A character class, or range, matches exactly one character of text, but the candidates for matching are limited to those specified by the class. Classes come in two flavors as described below:

- `[...]` Regular class, match only characters listed. - `[^...]` Negated class, match only characters *not* listed.

As with the dot token, by default negated character classes do not match newline, but can be made to do so.

The characters that are considered special within a class specification are different than the rest of regex syntax as follows. If the first character in a class is the ']' character (second character if the first character is '^') it is a literal character and part of the class character set. This also applies if the first or last character is '-'. Outside of these rules, two characters separated by '-' form a character range which includes all the characters between the two characters as well. For example, '[^f-j]' is the same as '[^fghij]' and means to match any character that is not 'f', 'g', 'h', 'i', or 'j'.

Anchors

Anchors are assertions that you are at a very specific position within the search text. NEdit regular expressions support the following anchor tokens:

- `^` Beginning of line - `$` End of line - `<` Left word boundary - `>` Right word boundary - `\B` Not a word boundary

Note that the \B token ensures that neither the left nor the right character are delimiters, or that both left and right characters are delimiters. The left word anchor checks whether the previous character is a delimiter and the next character is not. The right word anchor works in a similar way.

Note that word delimiters are user-settable, and defined by the X resource wordDelimiters, cf. X Resources.

Quantifiers

Quantifiers specify how many times the previous regular expression atom may be matched in the search text. Some quantifiers can produce a large performance penalty, and can in some instances completely lock up NEdit. To prevent this, avoid nested quantifiers, especially those of the maximal matching type (see below.)

The following quantifiers are maximal matching, or "greedy", in that they match as much text as possible (but don't exclude shorter matches if that is necessary to achieve an overall match).

- `*` Match zero or more - `+` Match one or more - `?` Match zero or one

The following quantifiers are minimal matching, or "lazy", in that they match as little text as possible (but don't exclude longer matches if that is necessary to achieve an overall match).

- `*?` Match zero or more - `+?` Match one or more - `??` Match zero or one

One final quantifier is the counting quantifier, or brace quantifier. It takes the following basic form:

- `{min,max}` Match from 'min' to 'max' times the previous regular expression atom.

If 'min' is omitted, it is assumed to be zero. If 'max' is omitted, it is assumed to be infinity. Whether specified or assumed, 'min' must be less than or equal to 'max'. Note that both 'min' and 'max' are limited to 65535. If both are omitted, then the construct is the same as '*'. Note that '{,}' and '{}' are both valid brace constructs. A single number appearing without a comma, e.g. '{3}' is short for the '{min,min}' construct, or to match exactly 'min' number of times.

The quantifiers '{1}' and '{1,1}' are accepted by the syntax, but are optimized away since they mean to match exactly once, which is redundant information. Also, for efficiency, certain combinations of 'min' and 'max' are converted to either '*', '+', or '?' as follows:

- `{} {,} {0,}` \* - `{1,}` \+ - `{,1} {0,1}` ?

Note that {0} and {0,0} are meaningless and will generate an error message at regular expression compile time.

Brace quantifiers can also be "lazy". For example {2,5}? would try to match 2 times if possible, and will only match 3, 4, or 5 times if that is what is necessary to achieve an overall match.

Alternation

A series of alternative patterns to match can be specified by separating them with vertical pipes, '|'. An example of alternation would be 'a|be|sea'. This will match 'a', or 'be', or 'sea'. Each alternative can be an arbitrarily complex regular expression. The alternatives are attempted in the order specified. An empty alternative can be specified if desired, e.g. 'a|b|'. Since an empty alternative can match nothingness (the empty string), this guarantees that the expression will match.

Comments

Comments are of the form '(?#<comment text>)' and can be inserted anywhere and have no effect on the execution of the regular expression. They can be handy for documenting very complex regular expressions. Note that a comment begins with '(?#' and ends at the first occurrence of an ending parenthesis, or the end of the regular expression... period. Comments do not recognize any escape sequences.

[eteran]

Yea, I am using the src/doc directory. Definitely feel free to fill in the blanks. I am making some changes from the source as I go (as you may have noticed).

1. I am using code markers for many things which are simply quotes in the source. Especially regex fragments.
2. I am replacing references to X-resources since that's no longer relevant.
3. Manually making links also refer to other pages in the wiki
4. removing the header/footer template code

[eteran]

I am curious about what formatting you saw that's missing. I looked, but am just not seeing it (probably because I'm not looking at it with fresh eyes like you are).

[anjohnson]

I didn't mean missing from your version, missing from my pandoc conversion because the original HTML is wrong. Things like the code markers that you mentioned - some places in the original use <code>'blah'</code> but should render as blah instead of 'blah' while others are missing the code markers completely. Many of those can be easily handled with a regex edit either before or after the conversion to gfm, which I will do.

[anjohnson]

FYI the converted help files are visible on my fork's wiki.

I'm not finished yet, I intend to merge with the changes you have made in your versions and fix things like replacing references to X Resources with the config.ini file. I love it that GitHub lets you clone a wiki with Git and work on it offline.

[eteran]

Fantastic!

[eteran]

@anjohnson Any update on this? Is your fork's wiki in pretty good shape to merge? I took a quick look and in general, it seems to be pretty good.

The only quirk is that I think we can do better than just the numbers for page titles.

Besides that, we can ditch a few sections too since they are pretty specific to the original nedit, most obviously would be the "Reporting Defects" section.

[eteran]

Oh, also, we may want to switch from a wiki to a "github pages" format (would be pretty easy to do). That would allow us to have a nice dedicated page with a table of contents sidebar. It would be very easy to switch, so you don't need to concern yourself with that detail :-).

[anjohnson]

@eteran Sorry I haven't worked on merging your changes yet, so most of the wording in my Wiki is still from the original text. I will look for some time in the next few days to get back to that.

The thing about GitHub's Wiki pages is that the page titles are taken from the Git filenames and hence from the page URLs. The original help text used the numbered pages for all the cross-links between pages, so to fix the titles will mean renaming everything and rewriting all of those internal links. By switching from a Wiki to GitHub Pages we should be able to avoid having to do that, so I think that's probably a better solution. I think it's better to keep the Wiki for developer documentation on building and translations without getting them mixed up with the help text.

Are you considering integrating the help into the application at all? I could see 3 ways to do that, in suspect increasing code complexity:

1. Have help bring up the user's browser pointing to the online help pages,
2. Add a Qt embedded browser window that displays the online help pages, or
3. Embedding the text into the application, with a Qt embedded browser window to render it.

These would probably affect how the pages should be formatted and where they might live, which is why I'm asking. I assume an embedded Qt browser would need to be fed HTML and not GitHub Flavored Markdown, so for option 3 at least the build system to generate the HTML would probably need some consideration. The other two would be able to work off the GitHub Pages site.

[eteran]

My main thought on help is that it'll be "online" The various help buttons will simply open the user's browser to the correct section of the help documentation. Essentially, your option 1 :-)

[anjohnson]

Okay, I thought you might say that, which is fine (except when the user is offline). One request would be to provide a configuration parameter that allows the base URL for the Help pages to be changed, for example to a local (file://) mirror.

I think I have now applied equivalent changes to the Regex pages that you made to your versions, and have also made various other updates.

Pages 28 and above are still heavily Nedit-specific though and may need you to do major surgery to their text and/or structure to remove things that don't apply to Nedit-ng. I changed the name and links to page 30 (X Resources) but haven't touched the text on it yet, it and the other customizing pages talk a lot about X resources files.

[eteran]

Sure, that's definitely something we can work in there. So do you feel it's in a state that I should try to set up the GitHub pages system for it yet? Or are you planning a little more work first?

[anjohnson]

I would like to make another pass through all of the pages checking for more formatting errors and minor updates (I didn't do all of them yet, hopefully over the weekend), but I don't want to hold you up. I don't know what kind of structure if any you're planning to add, but maybe you could start the gh-pages branch from the tip of my wiki repo (using a Subtree Merge say), which would allow you easily to pull any later changes I make into that branch. I think I read that GitHub Pages supports the use of git submodules, so that might be another approach to consider too, (start by pointing to my wiki repo and move it later).

One thing to consider early on is whether you might eventually need to have multiple versions of the documentation online simultaneously for different major releases say (so a version number would appear in the base URL that Nedit-ng uses). Submodules might help there too, you could define a separate git submodule for each version, pointing to different branches of the help repo.

[eteran]

I'll have to look into the subtree merge thing :-)

As for the versioning, maybe tags in that branch can handle that? Dunno, The documentation shouldn't need to change too often, so I wasn't too concerned about it. Though I do have some large preference changes planned, so that part will need to be rewritten eventually.

[anjohnson]

The versioning requirement means that all versions of the documentation will have to be online at version-specific URLs, so branch tags on a gh-pages branch won't help. However you could use git submodules or do multiple subtree merges of different version branches into different places in the gh-pages repo to implement versioning.

To support translations in the future the language name would also need to be a part of the URLs, but you could always introduce a new version of the documentation (adding the language to the URL paths) if you ever get any serious offers to translate the help text.

Question: Do you want me to replace NEdit with NEdit-ng throughout the text? I have just replaced the blurb at the top of the Getting Started (01) page, adding wording about the port to Qt from your README.md, but so far I haven't done any other name substitutions.

[eteran]

Once we get a first version up, I'll look into how to best deal with multiple versions. My initial plan is to use a "Read the Docs" template generated with mkdocs (like here https://www.mkdocs.org/user-guide/deploying-your-docs/).

I think I read somewhere that they support versioning with git tags, I'll double-check though.

Probably a good idea to replace NEdit with NEdit-ng throughout.

[eteran]

Translations are an interesting thought, I hadn't considered it even though it's pretty obvious. Probably can be handled nicely with a language sub-dir like:

https://eteran.github.io/nedit-ng/fr/
https://eteran.github.io/nedit-ng/es/

etc...

[anjohnson]

NEdit-ng currently only has UI translations for French and those do have to be compiled in, but the thing about documentation and online help texts is that they may be offered later for earlier releases, so it would be nice if the code can at least look for help in other languages in case it appears later.

It should also be possible to update the English version after the release, but a user should never see help that was written for a later release of the code than they are running. The updating requirement means you probably can't use a tagged version in the way that ReadTheDocs normally does because you want to be able to replace an already tagged version after an update. I might be wrong if they handle moving tags, but that would seem unusual.

Any, I have just pushed all my updates to my Wiki, enjoy!

What would you like me to do about the content and links on these pages:

• Key Bindings
• Version
• Mailing Lists
• Problems/Defects

It looks like many of the Action Routines are no longer supported, e.g. mark-letter(), keep-dialog(). I haven't checked the similar Macro Subroutines list, my guess is that you would might remember those changes faster than I would take looking up the routines.

[eteran]

That's a fair point about versions. We may need to use directories instead of tags, though I think we can tackle it after we have help at all ;-) I'll be sure to not make it so things in NEdit-ng's code makes too many assumptions about where the help is located once that part is added.

Regarding those pages, I'll take a look at keep/remove them as needed.

Regarding Action/Macro routines. classic NEdit supported both hyphens and underscores for functions names, but the hyphens were only there for legacy support. I think that NEdit-ng is a good place to chop that legacy support since it's a trivial conversion anyway. So yea, while mark-letter doesn't work, mark_letter should. and so forth.

I'll take a look at your wiki and start working on migrating it to a gh-pages branch probably tomorrow or tues, depending on my work load,

Thanks!

[eteran]

It's still in need of a little tweaking, but it's up! Check it out at:

https://eteran.github.io/nedit-ng/

I'll be playing with it a little more to clean up to little details, but going forward, it should be super easy to maintain :-D

(But yea, I still need to sort out version and language concerns, but this is a huge start, thanks so much for your efforts on this).

[anjohnson]

Looks great. Something's up with the title for 14 in the sidebar index, but you've probably seen that already (looks like it comes from a typo in the docs/mkdocs.yml file). I've found a few more places in the text where I could do some more formatting fixes, but I'm not sure when I'll get time for them now. I see you've put the help files in the nedit-ng/docs/ directory, presumably the best way to send you updates when I do get to them is as a PR against the program sources?

If you're going to drop the hyphenated function names (I've personally grown to rather liking them in Raku [the language formerly known as Perl 6] but I admit that their use isn't exactly mainstream and I haven't written any NEdit macros using them) we'll need to fix those names in the doc (in 24, 27 and some elsewhere too) and remove the Deprecated Functions section of 24.

[eteran]

Yea saw the issue with "Finding Declarations", it was indeed a typo. I'm going through and making small tweaks as I go. So there will be lots of improvements in the new few days.

Yes, definitely please make PRs against the master branch sources. the gh-pages branch simply has the final product of the documentation.

I agree about the hyphenated names, the thing with them is that - is also an operator, so the parser had to resort to some ugly hacks to support it. For example, is hello-world a function or is it the variable world being subtracted from the variable hello?

It was... just yuck :-/.

But yea, the docs will need to be updated to reflect this.

[eteran]

I have been going through the docs and making small tweaks as I go, about 10 pages done!

I am making some standardizing changes in formatting and grammar and making things clear, concise and consistent.

Menus will be referred to like this: Edit → Copy We don't say " Use the Edit → Copy menu item. We say "Use Edit → Copy".

Keyboard buttons will be referred to like this: Ctrl We don't say "press the Ctrl key" We say "press Ctrl"

Help section names won't be in quotes, and won't be referred to as "help sections". We don't say, "In the help section "Using the Mouse", ..." We say: "In Using the Mouse"

As I said, I'm about a quarter done, but these docs are starting to look great! Thanks for getting the ball rolling!

[eteran]

@anjohnson

OK, I've gone through every page and updated things to be consistent and clear. I'm probably mostly done with the actual changes the help text (I MAY do a general "grammar / spelling" check on it, but that's about it).

Of course, feel free to open any PRs regarding the help text as you see the need to. My next focus will be thinking about versioning support and then wiring up all the buttons in NEdit itself.

[eteran]

Versioning support is ready!

Translations may not be as easy, I'll have to look into that...

But now you can get help for a specific version with URLs like this: https://eteran.github.io/nedit-ng/2019.4/

NEdit-ng will be sure to use the version when refering to them in the code so that they are stable over time :+1:

[anjohnson]

Wunderbar! I think you got the last few bits I remember needed fixing; if I do come across anything I will submit a PR. Thanks!

BTW I love the way the yaml files turned out too, they're so much easier to read now.

[eteran]

Great to hear!

I'm going to close this now that the master branch has all the help buttons wired up. I'm sure that there is the potential for more improvement over time, but I think we've satisfied this issue.

Thanks again for your help!