Support zsh

[eigengrau]

So, I’ve noticed quite a while ago that zsh support had been removed. So, since christmas is a time for miracles, I thought I might check back on any recent developments. From searching the issue tracker I take it that nothing has changed in this regard. So I figured I might open this issue so that patient, interested parties can subscribe to it and get notified in case someone feels like stepping up.

That is of course if you do deem zsh support generally useful. Otherwise please feel free to close the issue.

Thanks for making shellcheck!

[HaleTom]

shellcheck is amazing, and it is the reason I still write most of my scripts in bash.

I'd love it if zsh support could be added though!

[molovo]

As someone with quite a few big ZSH projects out in the wild, I'd definitely be interested in seeing this happen. I'd be very happy to contribute myself if someone could let me know what would be required to get it implemented.

[bittorf]

@molovo: first you have to ask yourself if the features of 'zsh' are a subset of other shells or if zsh has own features. look into the source for 'first citizen shells' and the handling of 'dash' and 'bash' and you will get a feeling, what is needed...

[koalaman]

This would require:

1. Implementing all the syntax that is unique to zsh, to allow scripts to be successfully parsed (Parser.hs). This requires some care to avoid strange parse errors for other shells.
2. Extend the variable flow tracing code and similar to support these new structures, so that shellcheck can tell when variables are modified and such (AnalyzerLib.hs).
3. Fixing or extending any tests that are not correct for zsh (Analytics.hs, Checks/*.hs).
4. Adding new and interesting Zsh specific checks.

Ksh shares basically all its syntax with Bash, which is why it's still supported even though almost no one uses it. Zsh required a lot more work for a similarly small number of users, which is why I unfortunately wasn't able to keep it up.

[LukeShu]

If zsh is ever supported, it should recognize #compdef and #autoload as shebangs for zsh, as these are magic headers used in zsh autocompletion helpers.

[kgizdov]

@koalaman I understand that bash is way more popular that zsh, but it still has a huge community. Would it be appropriate if you make a project/branch for zsh contributions so that users can contribute. Maybe put a note in the README and helpwanted flag here. If you set up the infrastructure, I'm sure people will bunch up to make this happen. I am willing to spare some time on this.

[brother]

@kgizdov on the other hand a PR with the start is pretty much the same. I assume that if you gather some people to do the work either they are welcome to the project or they will fork it. The code won't be written and maintain by pure magic, as far as I can understand @koalaman will not take that responsibility himself.

[paulmelnikow]

I looked into this briefly a couple weeks ago. After I read through most of Parser.hs it was apparent that koalaman's step 1 would require deep knowledge of zsh syntax and bash syntax and a decent understanding of Haskell.

That made me wonder if zsh's parser (it's in C) could be used directly or modified slightly to emit an AST that could be interfaced to some high-level language constructs in Haskell or some other language. The wrapping is a bit involved, but is less work than reimplementing the parser.

I'm curious, has anyone on this thread worked with the zsh source?

[molovo]

@paulmelnikow I haven't worked with the source, but I have considered building an AST or something similiar (not for this purpose, but to support a coverage driver for zunit). I wonder if something could be built to tackle both issues at once?

[paulmelnikow]

@molovo Interesting idea!

[snoblenet]

Maybe the wiki could recommend a shellcheck alternative for zsh users? Because all Google searches based on variants of lint zsh take you back to shellcheck.

[paulmelnikow]

Unfortunately, there is no such equivalent. If anyone could make it happen, that'd be @molovo.

You may want to take a look at fish shell, which is better documented, more user-friendly, and developed on github.

[ghost]

I would like a support for Zsh but well Bash is still pre-installed in all major distro.

I code in Bash using Zsh...haha

[darthdeus]

I guess one could still use shellcheck for bash-ish zsh files? To be more specific, I have quite a few config files related to my ~/.zshrc and that's about the only place where I use zsh.

But being completely honest, I'm way more familiar with bash than with zsh, so sometimes when shellcheck is showing a warning, I'm not 100% sure if it applies to zsh as well.

[psprint]

I see Zsh thing happening if:

1. Shellcheck will be forked to e.g. ZShellcheck.
2. The authors do some effort to clean up the source to allow a fresh start for Zshell extensions, e.g. authors would disable support for ${!VAR} bash indirect-reference syntax and other things like this. It might be that some Bash property/feature required a hack or a bloated code somewhere, and this could be removed now.
3. A good code documentation will be available, and a channel to receive help from the authors.
4. Some people will get around the forked project, start small by e.g. implementing support for autoload functions (a feature of Zsh).
5. This way original shellcheck will sleep well, being undisturbed with Zsh-universe related actions.

I write this because I'm 3-years active Zsh (upstream) and Zsh-script-projects developer, and the above scenario covers my actual expectations to have a well organized environment/setup for working on a project.

BTW., Zsh can parse its code itself with a (z) substitution flag, I have based on it a Doxygen-like tool that works also with Bash scripts, check out significant-large Bash project (not mine) that uses it for Bash backend-scripts (the link points to a code-documentation generated with Zshelldoc). A code-documentation tool is somewhere in direction of linter tool.

[HaleTom]

There is so much commonality between {ba,z}sh that there would need to be a lot of importing upstream commits into the new project... @psprint What do you see as the advantages of a forked project?

Would it make sense to:

1. Move all common {ba,z}sh syntax to a meta-syntax, for example named: bazsh
2. Implement only the bash-specific extensions (eg things like ${var@Q}) in bash
3. Start the zsh-only syntax with a clean slate (importing only the common bazsh syntax)?

(I'm assuming one syntax can import/inherit from another syntax here - please correct me if I've misunderstood)

[psprint]

I think that without fork programmers will have to be real experts in Bash and Zsh. I learn Zsh for 3 years and sometimes there's still something that's new to me. If I imagine that I now would have to reach the same level in Bash but not through play & experiment like with Zsh, but with studying documentation, then I feel like running away from contributing.

My experience also tells me that structural programming doesn't match situation, where every action that a program has to make is also a result of condition like "Bash or Zsh". If I imagine codebase that's doing constant "Bash or Zsh" checking, I see something really severe, a mess. This mess can be controlled but this doesn't match Open Source programmers, IMO. Even well paid corpo programmers could fail and consciously doom project's codebase if someone would require of them that level of self control and care for details. Nobody would be able to prove them that they're making a crime for their comfort.

Forked project can just accumulate perfection in following Zsh syntax, it's a much simpler situation and it matches Open Source way of doing things.

[HaleTom]

I believe that the answer may depend on whether whether one shellcheck syntax-checker can inherit easily from another.

If it's possible, then perhaps something like this is possible:

1. Move all current bash syntax checkers into bazsh checkers
2. Create empty zsh syntax checkers inheriting from bazsh. These can be added to incrementally.
3. Allow bug reports to discover bash-only checkers in bazsh (and therefore in zsh), and move them to bash (which also inherits from bazsh.

This would hopefully avoid much of the shell version checking you describe.

I wonder how it is currently done for ksh vs bash that @koalaman mentions here?

@koalaman could you comment?

[danielbayley]

Wouldn't the simplest approach be to create a zsh branch or in the initial logic choose which parser to use based on the bang line #! /bin/zsh/#! /bin/bash etc…?

[psprint]

@HaleTom Ok the idea seems reasonable, it opens a way to do programming not whole-picture-grasping, which is pretty much the thing that I opt for. I think at this point, what's needed, is the Spellcheck authors to tell us how much the code is done in direct-structural way vs. indirect-with-high-degree-of-freedom way. I mean, you wrote about kind of syntax-"backends" that can be moved level-up or level-down between general-shell and specific-{bash,zsh} levels, to gradually separate things, but in reality I suspect this will have to be coded first, and I wonder how much parsing-knowledge or parsing-certainty/comfort would it require to actually approach this task.

[koalaman]

I appreciate your assumptions that ShellCheck is an enterprise level project with smart abstractions and a clever design, but you're really overestimating both it and me. Apart from being written in Haskell, ShellCheck is very simple and straight forward.

It essentially just has one parser and one set of checks that are applied to the result.

The parser handles all syntax for all shells the same way. If you use a Bash specific feature in Dash, such as for ((;;)), then it'll be parsed and interpreted as that Bash feature, and will get all corresponding Bash warnings. There's hopefully also a check that warns that it's unsupported in Dash, but that doesn't change how ShellCheck processes it.

There's no reason why you couldn't additionally recognize Zsh for x (1 2 3) and do the same. Note especially that it's not necessary to implement more than a single statement type at a time before you can run it and see the effects.

As for the checks, they don't have groups or stages. The only structure imposed is optional, and is purely for convenience and efficiency (like registering to handle a command name instead of having to iterate the AST to look for them). If a check is specific to a different shell, it'll just check and do an early return.

I doubt any of that would be a concern though. 95%+ of checks that apply to Bash also apply to Zsh (fgrep isn't less deprecated for example). There's probably <5 that don't, and those could just return early as well.

[psprint]

I see, from that point of view it looks easy. With the possible problems anticipated by me, I was thinking about things that are more subtle than an additional syntax. Like for example word-splitting being the default in bash, Zsh whole array referencing with $array instead of ${array[@]} (however: it works like in bash if ksharrays option is set), array indexing starting at 0 not 1, etc. I have had an impression that there is much such things, but I cannot be sure about this, would have to sit down and think about this a little longer.

[psprint]

@koalaman; Can it be said, that after implementing checks for a most easily-occurring mistakes, Shellcheck took a path of implementing checks for a more complex & rare, more specific, narrow, fine-grained mistakes? And that that's why the execution of a shell script has been implemented – to be able to dig into the states of the code and catch the fine-grained mistakes?

[koalaman]

ShellCheck does not support execution of shell scripts. Are you thinking of shfmt?

[psprint]

Ah, yes, shfmt. That said, I've finally have looked at the README. It surprised me positively because it listed the checks that are being done. I think that looking at the checks I can confirm that Shellcheck went from general scripting error checking (e.g. sh-word-split problems) to fine graned checks that e.g. target a syntax of a single command, like find.

[koalaman]

ShellCheck's original purpose was to auto-suggest relevant Wooledge BashFAQs and Pitfalls on IRC, so that's where most of the original checks come from. These include a mix of syntactic, semantic and command-specific issues.

[psprint]

Ahso, so maybe Shellcheck was fine-grained & specific from the start. I've got to the idea of fine-crafted, sophisticated-situations-targetted, when I thought "What checks I could actually do? Despite shwordsplit warnings on an unquoted variable use?" (I'm thinking about starting a Zsh linter project), and this lead me to the conclusion that I can target very specific situations but with possibly large number of them, making the linter a tool that often reports errors even though each of them is rare.

It essentially just has one parser and one set of checks that are applied to the result.

I think that if Shellcheck has a very large number of checks, then it is entering a more advanced level than the quote states, it's a level of knowledge base-like tool.

[rolfbjarne]

zsh will become the default shell in macOS later this year, so zsh will become much more common quite fast.

[LwsBtlr]

It will suddenly be installed an the default shell on tens of millions of machines.

[dpwolfe]

zsh will become the default shell in macOS later this year, so zsh will become much more common quite fast.

It was announced that it will release this Fall, which starts September 23rd: http://osxdaily.com/2019/06/18/macos-catalina-release-dates-public-beta-final/

In the meantime, if you upgrade an existing MacBook to the 10.15 Beta, you'll get this every time you open a new terminal session:

The default interactive shell is now zsh.
To update your account to use zsh, please run `chsh -s /bin/zsh`.
For more details, please visit https://support.apple.com/kb/HT208050.

I just set the flag in my .bash_profile to silence the warning for now until I get to porting my scripts:

export BASH_SILENCE_DEPRECATION_WARNING=1

[bp88]

zsh will be the default shell in macOS Catalina: https://support.apple.com/en-us/HT208050

[walles]

If @koalaman isn't super keen on maintaining zsh support, then maybe starting a new project for checking zsh scripts would be better.

On zsh's popularity, from https://insights.stackoverflow.com/trends?tags=bash%2Czsh. Let's see what happens once macOS Catalina is out though:

[psprint]

I'm starting a linter written in Zsh script. The point is: even if it will be a little slow (not sure how slow actually), it's worth trying as the effort is quite low, because Zsh exposes its parser via (z) substitution flag. The clue to make it work is to make it single–pass, I think, i.e. it should try to match all the rules at once, during a single token-loop cycle. I think that the ZQL language that I've developed for declare-zsh (https://github.com/zdharma/declare-zsh/wiki/ZQL-draft) is quite of use for the rules, especially the #,##,$,$$,… denotations.

[psprint]

Actually, I think that this should be a Rust project. It is now clear to me that it's easy to use the parser's C code from Zshell, and Rust allows to easily use C parts in its projects, so there's a very open way for the linter. The Zsh script that's already written will serve as a prototype :)

[rugk]

So what? Do you want to re-implement the whole shellcheck in Rust?

[psprint]

As the author said, shellcheck just applies the rules. After the parsing obstacle will be solved, the only thing that'll be left to do will be to write the rules for it. With the ZQL, that will be very easy.

[XVilka]

I remember Gentoo once started a project to parse Bash scripts in C - libbash. For some reasons it was abandoned, but maybe can be reused in such a rewrite:

• https://github.com/groundwater/gentoo-libbash
• https://wiki.gentoo.org/wiki/Google_Summer_of_Code/2012/Ideas/libbash

You might even go further and convert the bash/zsh sources from C to Rust using amazing c2rust automated converter and refactoring tool.

Wish you luck, that would be an amazing project!

[t-ricci-enhancers]

This is more relevant now that macOs Catalina uses zsh as the default shell

[JayBrown]

#!/bin/zsh
# shellcheck shell=bash
emulate -LR bash

😉

[rgoldberg]

#!/bin/zsh

# shellcheck shell=bash

emulate -LR bash

😉

@JayBrown This completely misses the point. zsh has a different syntax with useful features not available in bash, and we want that syntax to be checked by some tool. We don’t want to limit what we can do just to use a tool.

[JayBrown]

You do know what a wink emoji means, right?

[ssokolow]

You do know what a wink emoji means, right?

That your message was spam intentionally wasting everyone's time rather than innocent obliviousness?

[rgoldberg]

You do know what a wink emoji means, right?

I’ve seen people use wink emojis in different ways. One way is to indicate that they think they are stating something so obvious that everyone else is an idiot for not thinking of it themselves.

I gave you the benefit of the doubt that you thought you were being helpful, albeit it in a rude manner. Now I know that you were being rude by making an unfunny joke.

[mcandre]

One thing that ShellCheck could do with a few short lines more code is to at least identify common zsh configuration files. Whether ShellCheck proceeds to bail and refuse to parse, I don't care. I just think it should be smarter about what constitutes a zsh script vs. other languages.

So

• zshenv
• .zshenv
• *.zshenv
• zshrc
• .zshrc
• *.zshrc
• zlogin
• .zlogin
• *.zlogin
• zlogout
• .zlogout
• *.zlogout

would be simple enough to mark as these-are-zsh-scripts.

[HaleTom]

@mcandre's solution would cause me real-world negative impact.

Here are the first 3 lines from my .zshrc:

#!/bin/bash
#      ^----- get shellcheck hints based on bash
# https://github.com/koalaman/shellcheck/issues/809

I do this because I prefer to have shellcheck's safety and occasionally eschew some functionality of zsh.

The above lines enable shellcheck, and I only disable it as needed when I don't know how to avoid zsh syntax.

Examples:

# shellcheck disable=2154,2004  # zsh
(( ${+functions[enable_you_should_use]} )) && enable_you_should_use

# shellcheck disable=SC2154,2034,2004
(( ${+_comps} )) && _comps[zinit]=_zinit

You can view my .dotfiles repository for more.

[mcandre]

As a workaround, I am using stank to filter out zsh-interpreted scripts before ShellCheck sees these file patterns.

stank -exInterp zsh . | grep -v node_modules | xargs shellcheck

[RunningDroid]

Here's an update to @walles' chart now that Catalina's been out for a few months:

[bl-ue]

Mid-2021 but zsh is still only 5% 🤔

[vitorgalvao]

Mid-2021 but zsh is still only 5%

Percentage of StackOverflow questions is a poor metric for this. Bash and ZSH are the same in most regards, so anyone looking for an answer to the latter will likely solve it with an example in the former.

Which means the biggest reason for ZSH questions are cases where it differs from Bash. Since those are highly specific cases, there’s no reason questions should suddenly spike just because it became the default on macOS.

[phdoerfler]

It's a poor metric in general. All it shows is that bash raises more questions than zsh (on Stack Overflow). If anything you could use that to make the point that zsh is less obscure than bash but for the reasons @vitorgalvao outlined even that makes not much sense. Speaking of bad scripting languages: I've seen a similar metric used to underline how much more popular Python was compared to R and the only thing that does is to show how much more questions Python raises over R :P