Many issues with bash / shell script highlighting

[okdana]

I'm not sure if it would be better to break these up into separate issues, but i wanted to err on the side of not spamming. Sorry if this is too general.

The Shell Script (Bash) highlighter is super super limited and in many cases broken entirely. In particular:

Braces in parameter expansion/interpolation are handled naïvely; any closing brace following the first opening brace (regardless of context) can prematurely close the expanded parameter, potentially breaking the rest of the file.

Similarly, parameter-expansion operators are handled naïvely — basically if any character used as an operator appears anywhere on the line after the variable name, regardless of context, it's highlighted as an operator. This includes operators that can only appear at the beginning of the expansion, like !.

The 'body' of a parameter expansion (the part after the operator) doesn't 'include' (in the sense of the tmLanguage file) other scopes.

^ and , (upper-case and lower-case, respectively) aren't recognised as parameter expansion operators.

Variable declarations (arguments to export, declare, typeset, local, readonly) and un-declarations (unset) are not highlighted.

Variable assignments are not highlighted (interpolated variables and variables like for i in ... are).

Arithmetic constructs are highlighted as a literal strings, except when they appear in double-quotes (or in a for loop). Edit: Looks like the theme probably just needs rules to make it not like this. I sort of question the scoping of these constructs as 'strings' though, especially the (( ... )) one.

Command-substitution constructs are highlighted inconsistently. Dollar substitution ($( ... )) is highlighted correctly, but the body of a back-tick substitution is highlighted as a literal string. Edit: This may be a function of the theme i was using, sorry. See also edit above.

Process-substitution constructs are not highlighted correctly (similar to back-ticks). Edit: Probably theme-related, see also edits above.

Nested interpolations aren't highlighted correctly.

The <<- operator (heredoc ignoring leading tabs) is not highlighted correctly.

Heredoc bodies aren't highlighted as strings (as they are for other languages).

Related to the above, the highlighting doesn't understand the effect of single quotes on a heredoc delimiter.

Literal patterns following a regular-expression operator are not highlighted correctly (should be string or dedicated regular-expression scope).

Operators in single brackets aren't highlighted correctly.

The in control key word (in case ... in) isn't highlighted correctly.

The break control key word isn't highlighted correctly.

Semi-colons aren't recognised as operators in for-loop arithmetic constructs.

The appearance of a control key word in a substitution construct will break the construct and potentially all following text.

I suppose shell scripts are not very sexy but it seemed useful to at least document these somewhere. Also, i'm sorry if any of these are theme-specific; i tried to rule those out but i may have got distracted trying to write the examples.

[FichteFoll]

Thanks for the very detailed report. I suspect the bash syntax will be rewritten anyway shortly, following http://www.sublimetext.com/forum/viewtopic.php?f=2&t=17892, so this will surely be of use.

[okdana]

Oh, yeah, i did hear something about that. Thanks for responding!

[leesei]

@okdana can you post the file you used to validate the syntax?

[leesei]

I discovered an issue with heredoc:

cat <<EOF
works fine without space
EOF

cat << EOF
with space it is not ok
EOF

(I noticed Github have the same issue after posting.)

BTW, when looking the syntax file, I see that many other syntax (like HTML and MARKDOWN) can be highlighted within the content of heredoc. How to trigger them?

[okdana]

I don't have any 'real-world' files i'm comfortable sharing (all stuff from my job), but here's a test file i put together quickly:

https://gist.githubusercontent.com/okdana/45d3fd437e2f97f2ac63/raw/a822842edf8acdb7af4d182bbe18545f36d4c7be/bashsyntax.sh

[leesei]

Thanks a lot. I think the file will help improve any Bash syntax highlighter.

Would you also add the Heredoc white space issue I posted to your Gist? (Gist has no PR :grimacing:)

[okdana]

Updated, sorry. That actually explains why i was seeing some of the behaviour i was, i didn't notice before.

I also noticed that the syntax thing does recognise the <<- operator (Ctrl+Shift+P reports string.unquoted.heredoc.no-indent) when it's not followed by white-space, though it still highlights the - incorrectly.

[jfcherng]

I am just trying to fix this issue. PR: https://github.com/sublimehq/Packages/pull/47

[okdana]

That looks awesome, thank you! I will definitely try it out as soon as i'm able.

[keith-hall]

note this repo has been deprecated in favor of https://github.com/sublimehq/Packages/issues/434

[FichteFoll]

The syntax definition has been rewritten via https://github.com/sublimehq/Packages/pull/1057. I imagine that most, if not all, of this has been fixed. I tested a few test cases manually but am too lazy to type out all of them (since they weren't provided as verbatim text).

If you find any remaining issues with the current bash/shell script highlighting, please open a separate issue on https://github.com/sublimehq/Packages (after confirming it doesn't exist already).