syntax: consider zsh support

[caarlos0]

Congratz for the great tool!

Would be nice to have zsh support.

[mvdan]

Zsh is a massively complex shell - even more so than Bash - so let me ask a few questions first.

Why? Is there much Zsh code in the wild? From what I can tell, Bash, POSIX Shell and mksh already cover pretty much everything.

Is there a language spec for Zsh? If there's none, is there a document that summarizes all the language features?

The bottom line is that I fear this would be an insane amount of work for little benefit. I never use Zsh, so there's also that - I have no personal incentive to do it.

Also /cc @D630 who suggested zsh alongside mksh (the last one I did, it wasn't much work on top of Bash/POSIX)

[mvdan]

Also note that I would put Zsh in a similar spot as Fish. They're cooler, newer shells that - as far as I know - are popular as interactive shells. But not so much for scripts that you maintain over time. Those tend to be in Bash or POSIX Shell, since you want the scripts to run on as many systems as possible. This is the kind of shell code that shfmt is for.

[caarlos0]

Also note that I would put Zsh in a similar spot as Fish. They're cooler, newer shells that - as far as I know - are popular as interactive shells. But not so much for scripts that you maintain over time. Those tend to be in Bash or POSIX Shell, since you want the scripts to run on as many systems as possible. This is the kind of shell code that shfmt is for.

yeah, I agree.

[incognitoRepo]

hello, no disagreements here. but I saw a very similar response in the equally "dam this is useful" shellcheck repo (im coming from vscode). now, im no shell expert, but i always had the impression zsh was the #2 game in town. at least, anyone who seems to care about aesthetics is almost guaranteed to be on zsh. its actually a bit disappointing to learn that its not that important (i.e., major tools don't support it). and really, i guess thats more on zsh people (who could make a spec or a pull req or smth). anyhow, thats just for context. great package!

[mvdan]

@incognitoRepo please note my earlier point about interactive shells. zsh may be popular as an interactive shell, but this tool is for scripts and programs one maintains. Can anyone provide proof that zsh is popular as a scripting language too? I've personally never come across a script in it, but I might be biased.

That's just as far as utility goes; after that, we'd also have to consider how much work it would be. I probably wouldn't have the time to impement all of its syntax features, particularly since bash support isn't even complete yet :)

[texastoland]

As of macOS Catalina (the current version) it's the default shell for new user accounts. It may result in more requests. I agree it won't increase real-world usage though.

Is there a language spec for Zsh? If there's none, is there a document that summarizes all the language features?

There's a pretty hefty manual (PDF) that I mainly reference when I see a glob or expansion I'm not familiar with. I couldn't find any formal grammar at all.

[mvdan]

Huh, interesting, I didn't know Mac had switched over. I'm not sure if that will result in people writing more zsh scripts, though.

[mvdan]

Different people have pinged me about the macOS move in the past few weeks, so let's reopen this issue for now.

Pros:

• Supporting sh, bash, and mksh, it feels like zsh is the next logical step by popularity
• Being the default shell of new macOS versions should mean that macOS users would start writing more scripts in zsh instead of bash
• zsh is pretty similar to bash/mksh, so syntax-wise, we should be able to reuse a good amount of code

Cons:

• Like bash, zsh is a very complex shell. bash work started nearly three years ago, and even today I still find bugs or missing features in the syntax package.
• Like bash and mksh, zsh does not have a formal language specification. This will make it harder to map out what language features we're missing.
• Increase in complexity, which would hurt binary sizes, compile times, and my ability to maintain this project long-term.
• It's unlikely I would ever have the time to write proper support for zsh in the interpreter package.

I'm honestly about 50/50 on this right now. Thoughts welcome; please keep it concise so that the thread doesn't derail.

[theclapp]

I used to use zsh daily. (More and more now I use mvdan/sh/interp! :) I wrote some scripts in it, almost entirely for my own use. My workplace(s) for years have generally been bash shops, so anything I wanted to share, I felt like I had to either heavily comment / document it, or just translate to bash. (In fairness, that wasn't usually a huge burden, but it was usually non-zero, nevertheless.)

So I like zsh, don't get me wrong!

But I also agree that it's very large, and kind of ad-hoc. It does kind of feel like there's an underlying logic to it, but that can be hard to suss out / reverse engineer.

I think it'd be fair to say "zsh maintainer wanted", and maybe implement often-used, low-hanging fruit, as time, demand, and contributions allow. (I was going to mention ** as a feature in zsh I miss a lot in bash, but I checked the bash manual and it turns out bash can do that, so I guess the joke's on me there. :)

Have you considered inquiring on the zsh mailing lists for collaborators?

[mvdan]

But I also agree that it's very large, and kind of ad-hoc.

The good news here is that I'm only considering parser and printer support, i.e. just the syntax. Zsh is different from Bash in lots of ways when it comes to running a script, but we don't care because I don't think it would ever be a good idea to add a "zsh mode" in the interpreter.

implement often-used, low-hanging fruit, as time, demand, and contributions allow.

This would definitely be a best-effort basis :) That's how we're doing Bash too. It would be insane to try to reach 100% syntax compatibility in a single release.

Have you considered inquiring on the zsh mailing lists for collaborators?

That's a good idea. I guess I can phrase it as an FYI, to see if there would be interest from any zsh developers or heavy users.

[texastoland]

I keep using shfmt and shellcheck with my zsh files. Most of the differences are semantic rather than syntactic. Here's an obtuse yet idiomatic example that doesn't parse:

#                ┌ nesting
#                │  ┌ flags ┌ modifiers
#                ▼  ▼       ▼
export ZDOTDIR=${${(%):-%x}:P:h}
# ...globbing also has additions

I see three potential levels of support:

1. Encourage encapsulation of unparseable code in separate files. It hasn't been too inconvenient for personal scripts. I'd write larger scripts in a different language anyway.
2. Support some comment to skip parsing the next line.
3. Only support differences at the highest level. My example highlights all the additions to expansions. You could test against one of the frameworks for edge cases.

[mvdan]

Encourage encapsulation of unparseable code in separate files.

That's pretty much what we have today though, no?

Support some comment to skip parsing the next line.

I don't think we want to do this. The parser should just parse code statically, without understanding special comments or anything like that.

Only support differences at the highest level.

I'm not sure if I understand. If we support zsh syntax, we should aim at supporting all of its documented syntax, just like we do with bash.

[texastoland]

That's pretty much what we have today though, no?

That's what I meant.

I don't think we want to do this.

To me it's the simplest and friendliest solution in the meantime.

The parser should just parse code statically, without understanding special comments or anything like that.

I didn't understand about statically. It's essentially like a comment or heredoc but ending after the first non-comment line instead of some token.

If we support zsh syntax, we should aim at supporting all of its documented syntax, just like we do with bash.

For instance expansions can start with flags in parentheses (${(%)...} in my example). Those flags can take arguments themselves separated by potentially arbitrary characters similar to pattern expansions. Rather than fully suss it out you could leave everything in those parentheses like a "todo" in the AST. That might lower the barrier to a minimally viable solution.

[texastoland]

I thought the only syntactic differences were related to expansion and globbing but I recently stumbled on code in the wild using alternate syntax for control structures.

[texastoland]

• Support some comment to skip parsing the next line.

I ended up implementing this with a wrapper script:

#! /usr/bin/env zsh

# match: # no-parse: explain here
typeset pattern='^\s*#\s*no-parse.*\s+'

# comment next lines
fastmod --accept-all "$pattern" '$0#' "$@"
shfmt -w -s "$@"
# revert commented lines
fastmod --accept-all "(${pattern})#" '$1' "$@"

[ajeetdsouza]

While there may not be as many small zsh scripts in the wild, there are definitely far more plugins, which are large codebases written in pure zsh. I think they would stand to benefit the most from a formatter. Some of the ones I use:

• powerlevel10k - 20k+ lines of zsh
• fast-syntax-highlighting - 1.5k+ lines of zsh
• ohmyzsh framework - 20k+ lines of zsh

I already use shfmt for zoxide's bash and posix plugins, I would love to use it for zsh as well.

[mvdan]

I'm strongly leaning towards implementing zsh support in the syntax package (and shfmt). The only question is allocating the 2-3 weeks of continuous work to get to a working prototype :)

If you would like to help by testing in the future, please react with the "eyes" emoji on this comment and I'll be in touch.

[docwhat]

IIRC zsh itself can dump out an abstract syntax tree. That could be useful for testing.

[mvdan]

@docwhat very interesting - can you share how to do that?

[docwhat]

The project I first learned about it was zdharma/zinit ... however, the owner deleted everything. It's moved to zdharma-continuum (yay! opensource!) but I'll have to dig around again to figure it out.

zinit uses the zsh parser to allow it to use syntax like this:

# All the same:
zinit --as=something
zinit as"something"

[docwhat]

Ah, here we go...

In zshexpn under Parameter Expansion Flags:

z Split the result of the expansion into words using shell parsing to find the words, i.e. taking into account any quoting in the value. Comments are not treated specially but as ordinary strings, similar to interactive shells with the INTERACTIVE_COMMENTS option unset (however, see the Z flag below for related options)

Note that this is done very late, even later than the (s) flag. So to access single words in the result use nested expansions as in ${${(z)foo}[2]}. Likewise, to remove the quotes in the resulting words use ${(Q)${(z)foo}}.

Z:opts: As z but takes a combination of option letters between a following pair of delimiter characters. With no options the effect is identical to z. (Z+c+) causes comments to be parsed as a string and retained; any field in the resulting array beginning with an unquoted comment character is a comment. (Z+C+) causes comments to be parsed and removed. The rule for comments is standard: anything between a word starting with the third character of $HISTCHARS, default #, up to the next newline is a comment. (Z+n+) causes unquoted newlines to be treated as ordinary whitespace, else they are treated as if they are shell code delimiters and converted to semicolons. Options are combined within the same set of delimiters, e.g. (Z+Cn+).

Example

#!/bin/zsh

read -r -d '' codetext <<'CODE'
# A function to help with diction.
the_rain() {
  local rain="falls mainly in the plain"
  echo "in spain ${(qq)rain}"
}
# end of the_rain
CODE

declare -r -a codearr=( "${(Z:c:@)codetext}" )

declare -i c=0 i=0
declare last=';'
for item in "${(@)codearr}"; do
  [[ $item == '{' ]] && ((i++))
  [[ $item == '}' ]] && ((i--))
  if [[ $last == ';' ]]; then
    for i in {0..$i}; do echo -n '  '; done
  fi
  if [[ $item == ';' ]]; then
    echo
  else
    tput setaf $((c + 1))
    echo -n ">${item}< "
    c=$(( ((c+1) % 7)))
  fi
  last=$item
done

echo

If you run it you get this (except the colors don't show up here):

  ># A function to help with diction.< 
  >the_rain< >()< >{< 
    >local< >rain="falls mainly in the plain"< 
    >echo< >"in spain ${(qq)rain}"< 
  >}< 
  ># end of the_rain< 

[marlonrichert]

@docwhat @mvdan

There's no need to parse Zsh code to be able to format it. Zsh can actually format it for you! 🙂

If you put the code into a function, then functions -x<tab size> <func name> can be used to print out fully formatted code, minus any comments or blank lines. For example:

% setopt interactivecomments
% tmp() {
#!/bin/zsh

read -r -d '' codetext <<'CODE'
# A function to help with diction.
the_rain() {
  local rain="falls mainly in the plain"
  echo "in spain ${(qq)rain}"
}
# end of the_rain
CODE

declare -r -a codearr=( "${(Z:c:@)codetext}" )

declare -i c=0 i=0; declare last=';'
for item in "${(@)codearr}"; do
[[ $item == '{' ]] && 
    ((i++))
[[ $item == '}' ]] && 
    ((i--))
if [[ $last == ';' ]]; then
for i in {0..$i}; do echo -n '  '; done
fi
if [[ $item == ';' ]]; then echo; else
tput setaf $((c + 1))
echo -n ">${item}< "
c=$(( ((c+1) % 7)))
fi
last=$item
done

echo
}
% functions -x2 tmp
  read -r -d '' codetext <<'CODE'
# A function to help with diction.
the_rain() {
  local rain="falls mainly in the plain"
  echo "in spain ${(qq)rain}"
}
# end of the_rain
CODE
  declare -r -a codearr=("${(Z:c:@)codetext}") 
  declare -i c=0 i=0 
  declare last=';' 
  for item in "${(@)codearr}"
  do
    [[ $item == '{' ]] && ((i++))
    [[ $item == '}' ]] && ((i--))
    if [[ $last == ';' ]]
    then
      for i in {0..$i}
      do
        echo -n '  '
      done
    fi
    if [[ $item == ';' ]]
    then
      echo
    else
      tput setaf $((c + 1))
      echo -n ">${item}< "
      c=$(( ((c+1) % 7))) 
    fi
    last=$item 
  done
  echo
%

After that, you can use the output of ${(Z+C+)…} to fine-tune the formatting:

• for each line indentation that's not internally part of a semantic token (so, for example, not in a << statement), decrease the indentation by one level,
• replace each <line break><indentation>[do|then] with ; [do|then],
• reinsert each line break that occurred in the original after a && or || and add indentation equal to previous line's indentation plus two,
• wrap too long lines (preferably after a && or ||) and add indentation after each inserted line break equal to the original line's indentation plus two, and
• figure out where to reinsert comments and blank lines.

[denysdovhan]

Hey, I'm the author of @spaceship-prompt (almost 17K stars and 12K lines of code). Looking forward to using shfmt against my codebase 🙂

[yutkat]

I found this one. https://github.com/lovesegfault/beautysh

[marlonrichert]

@yutkat Doesn't look like that has Zsh support either, though.

[yutkat]

Although support is not explicitly noted, but it worked fine in my zsh program.

[0xdevalias]

Currently the main/only thing i've noticed shfmt breaking within my zsh scripts is this sort of syntax:

-if (( $+commands[git] ))
-then
+if (($ + commands[git])); then

Where the $ + command (instead of $+command) then becomes invalid zsh script, and causes the script to break.

[gibfahn]

The above formatting is weird but not the end of the world, but I find this syntax tends to cause shfmt to error completely and not format the file:

project_dir=${0:a:h:h}
cd $project_dir

❯ shfmt -w foo.zsh
foo.zsh:13:20: ternary operator missing ? before :

[nimish]

There's definitely some bogus problems raised when dealing with zsh:

"${XDG_CACHE_HOME:-$HOME/.cache}/p10k-instant-prompt-${(%):-%n}.zsh"

Is valid zsh but has a spurious parameter expansion error.

Given macOS is shifting to zsh, I have been encountering more and more zsh-isms.

[pboushy]

I'm a "MacAdmin" working at a large company. Our rule is if you can write the script easily in POSIX sh, do that. If your script need arrays or anything complex use ZSH.

I know many other companies are doing the same.

ScriptingOSX.com is one of the main websites for learning how to script on macOS, and all his latest stuff is ZSH.

Adding ZSH support would be awesome.

I mentioned to the shellcheck maintainer that I'd be willing to contribute if they can provide some guidance on bite-size pieces to tackle first. But then I discovered it's written in Haskell... which requires learning an entirely new language for me on top of taking on a complex enough area already. While I don't have a lot of experience with go, I'm not brand new to it.

[rseichter]

Here's an example of a valid ZSH statement which is not valid for BASH. The statement is part of my .zshrc and works fine there.

. "$XDG_CACHE_HOME/p10k-instant-prompt-${(%):-%n}.zsh"

shfmt -w filename returns the error "parameter expansion requires a literal". Out of curiosity, I also tried shellcheck -s bash filename, resulting in error SC2296.

I use ZSH on all my macOS and Linux based machines, and I would appreciate ZSH support in shfmt. Thanks.

[jansorg]

After implementing initial Zsh support for BashSupport Pro I can confirm that it would be a huge amount of work to implement Zsh support for shfmt. Most notable differences I've found:

• Parameter expansion flags, e.g. ${(aO)values} or even with additional arguments ${(g:opts:)name}
• Modifiers, e.g. $relativePath:a which are used for globs and variables.
• Alternate forms of complex commands, e.g. while list { list }
• Subtle differences in syntax support, e.g. command separators in blocks are optional { myCommand } instead of Bash's { myCommand; }
• Extensions like multiple names passed to function definitions function a b c { body; }, anonymous functions like function { local var="only-here"; echo $@;} a b c
• Extended arithmetic operators, e.g. &&=
• The "Zsh amper bang", e.g. a &| b
• and a bunch more

tl;dr Zsh is an even more complex language than Bash and adding support would take a lot of time

[mvdan]

Fully agreed with @jansorg, even more so because both bash and zsh are languages that work "as documented and implemented" without a formal spec, so they tend to have undocumented behaviors and evolve rapidly to add new features. At the same time, shfmt can work OK for many people in practice with partial support for the most commonlly used features. Just look at the earliest releases of this software - bash support was very limited and buggy, but some people still used it.

I also wanted to give signs of life, as I haven't given an update in over two years. This is very much still planned, but I'll keep quiet until I find the time to work on a prototype, because it doesn't feel right to make promises on ETAs.

[rseichter]

I am certain that adding ZSH support is not an easy task, and my previous comment was mostly meant to indicate that I am a member of the potentially growing group of people who would be grateful if shfmt supported the ZSH language. No pressure, just something on my wish list worth mentioning. 😉

[mvdan]

No worries, this thread and the added comments aren't causing me any sort of stress right now. I just noticed it's already been over two years since I last gave an update (geez, already?) so I wanted to briefly update again.