RegExs Syntax Highlighting

[tajmone]

The [[insert_code]insert_code node has two attributes which contain regular expressions: from_regex and to_regex.

It would be nice to syntax highlight the RegEx string according to RegEx semantics. ST already provides RegEx syntax highlighting for many language (Ruby, etc.) which makes reading and writing RegExs much easier, and is also a valuable guide in ensuring that a RegEx is correct (e.g. escaping edge cases, and so on).

We could achieve this by passing the whole RegEx string to an embedded syntax — either our won custom RegEx syntax, included in the package as a hidden syntax, or using a native ST4 RegEx syntax (compatibility permitting):

• https://github.com/sublimehq/Packages/tree/master/Regular%20Expressions

Obviously, if we could just re-use ST's native RegEx syntax it would save us a lot of work, but we need to ensure that the syntax is compatible with that of PML's RegExs!

@pml-lang, I assume these RegEx use Java's native Regular Expressions. Is it so, or you're using some custom engine (e.g. PPL's own RegEx)?

Could you provide me some info on Java RegEx syntax used here, and some links to documentation, etc.?

Any tips and advise on this feature?

Reference Links

• sublimehq/Packages — Sublime Text native packages repository.
• Regular-Expressions.info:
  • Java Regular Expressions
  • Features comparisons between engines and langs
• Regular Expression Engine Comparison Chart — Gist by @CMCDragonkai (Roger Qiu).
• Java — Java doesn't have a built-in RegEx class, but there's the java.util.regex package:
  • Regular Expressions and the Java Programming Language — at Oracle.
  • Java Regular Expressions — at W3Schools.
  • Oracle Docs:
    • Java SE/JDK 15 » java.util.regex
    • Class Pattern
  • raimonbosch/java.pcre — Java implementation of PCRE, by Raimon Bosch (@raimonbosch).

[tajmone]

ST's Native RegEx Syntaxes

The sublimehq/Packages repository contains the following RegEx syntax definitions in the `

• RegExp.sublime-syntax — Matches PCRE's RegEx syntax, used in various langs and in ST's Find panels.
• Regex Replace.sublime-syntax — Regular Expression Replacement. See:
  • ST Docs » Syntax Definitions
  • Boost-Extended Format String Syntax
• File Pattern.sublime-syntax — See ST Docs » File Patterns.

We might end up simply embedding one of these syntaxes, readapt them, or use them as inspirational material; in any case, they're going to be useful for the task at hand.

They are released under a very permissible license:

If not otherwise specified (see below), files in this repository fall under the
following license:

    Permission to copy, use, modify, sell and distribute this
    software is granted. This software is provided "as is" without
    express or implied warranty, and with no claim as to its
    suitability for any purpose.

An exception is made for files in readable text which contain their own license
information, or files where an accompanying file exists (in the same directory)
with a “-license” suffix added to the base-name name of the original file, and
an extension of txt, html, or similar. For example “tidy” is accompanied
by “tidy-license.txt”. © 2021 GitHub, Inc.

[pml-lang]

I assume these RegEx use Java's native Regular Expressions.

Yes, I confirm.

Could you provide me some info on Java RegEx syntax used here, and some links to documentation, etc.?

The full syntax is documented in Java's Pattern class.

[tajmone]

The full syntax is documented in Java's Pattern class.

Thanks for the link. I've added it to the Ref. Links section above.

Although I don't know Java RegEx flavour in detail, from what I've gathered is mostly similar to PCRE, but lacks some of its feature. I also noticed that many people ask on forums how to use PCRE in Java apps.

I was wondering, would it be hard or inconvenient to use PCRE for PML? In the above links I added a link to a repository with a PCRE implementation for Java:

https://github.com/raimonbosch/java.pcre

I believe that PCRE is the most known RegEx flavour, so it would be good if PML used that for its RegEx.

---

PS: As a side note, I wanted to ask you how the Sublime PML package updates are unrolling for you on ST4. Since this is just a repository, and not a package published on Package Control, I'm not sure how end users are getting the updates (e.g. they happen silently in the background, or you get a notification?).

In the past days I added some new features and fixed some bugs, but I guess end users aren't aware of them since I can't use the update messages for the package, because the repository doesn't yet have release tags. Once it reaches a stage where I can associate a tag with each release update, then I could have ST open a custom text file whenever a new version was installed, mentioning the new features.

[pml-lang]

I was wondering, would it be hard or inconvenient to use PCRE for PML?

To keep things simple, PML only depends on standard Java. No third-party libraries/frameworks are used. I want to keep it that way, unless there is a really good reason to add a dependency. As for regexes, I think they will be used rarely in PML, and if they are used, it's probably just simple regexes that don't require features not supported by the standard Java regexes. However, I could be wrong, in which case we can reconsider using another regex engine.

how the Sublime PML package updates are unrolling for you on ST4

There is no automatic update on my PC. I think that updates must be done manually.

[tajmone]

There is no automatic update on my PC. I think that updates must be done manually.

The native Package Control package should update the Sublime PML repository clone automatically, provided that the Git binary is available on the System PATH. The problem is that this would be a silent operation in the background, with no reports, so unless there's an obvious change in the package behavior you won't be aware of it.

The only way to enable package update messages showing up in ST is to create tagged releases, which I was trying to avoid since it's all pretty much Alpha stage work right now. I could though create Alpha releases/tags, and just delete them in the future when the package is released on Package Control.

If end users are not made aware of new package features, it's unlikely they'll be using them and giving feedback (which is quite important in this stage).

The only alternative solution I can think of is including a features changelog document in the package, and making it accessible via ST menus, so that it can be viewed by end users to keep up with new features.

[pml-lang]

provided that the Git binary is available on the System PATH

That's the case on my PC. C:\Program Files\Git\cmd is included in PATH.

including a features changelog document in the package

That would be nice. And still useful later too, when updates are done automatically via ST's package manager.

[tajmone]

That would be nice. And still useful later too, when updates are done automatically via ST's package manager.

Yes, I should add a changelog of sorts, listing significant changes that might affect end users (new key shortcuts, featutes, etc.), maybe leaving out the tech nitty gritty behind the scenes.

But the point is that package should already be updated via ST's Package Manager — and if it's not, then there's probably something wrong with custom settings, or you've cloned your own fork instead of the upstream.