Performance on VERY large script

[maximunited]

For bugs

• Rule Id (if any, e.g. SC1000): any
• My shellcheck version (shellcheck --version or "online"): 0.9.0

Here's a snippet or screenshot that shows the problem:

#!/bin/bash

$ wc -l myscript.sh
10370

The problem is that shellcheck is taking ~12GB of RAM and sometimes CPU reaches 100% when I edit the script w/ VSCode. I really love the onType option, but it kills my laptop.

Can the shellcheck run partially on specific line range instead of all file?

[aeiplatform]

There must be something wrong with your environment because when you count all the characters in the script. Realize how huge 12GB is. I run checks on 2k lines with raspberry pi like instantly.

[aeiplatform]

shellcheck needs to know the entire source code for reference etc

[aeiplatform]

Try running /usr/bin/time -v strace -cf shellcheck your-script.sh for more info.

[aeiplatform]

If is this just happening in your IDE plugin then I think this is wrong repo to look for answers. VS Code Shellcheck

[maximunited]

--- Process 44976 created
--- Process 44976 loaded C:\Windows\SysWOW64\ntdll.dll at 771F0000
--- Process 44976 loaded C:\Windows\SysWOW64\kernel32.dll at 75270000
--- Process 44976 loaded C:\Windows\SysWOW64\KernelBase.dll at 760C0000
--- Process 44976 loaded C:\ProgramData\Symantec\Symantec Endpoint Protection\14.3.5427.3000.105\Data\Sysfer\x86\sysfer.dll at 75000000
--- Process 44976 thread 34476 created
--- Process 44976 loaded C:\Windows\SysWOW64\shell32.dll at 763E0000
--- Process 44976 loaded C:\Windows\SysWOW64\msvcp_win.dll at 769A0000
--- Process 44976 loaded C:\Windows\SysWOW64\ucrtbase.dll at 759A0000
--- Process 44976 thread 27480 created
--- Process 44976 loaded C:\Windows\SysWOW64\user32.dll at 75AD0000
--- Process 44976 loaded C:\Windows\SysWOW64\win32u.dll at 75FB0000
--- Process 44976 loaded C:\Windows\SysWOW64\gdi32.dll at 770E0000
--- Process 44976 loaded C:\Windows\SysWOW64\gdi32full.dll at 75EC0000
--- Process 44976 thread 11724 created
--- Process 44976 loaded C:\Windows\SysWOW64\imm32.dll at 75FD0000
--- Process 44976 loaded C:\Windows\SysWOW64\sechost.dll at 76A60000
--- Process 44976 loaded C:\Windows\SysWOW64\rpcrt4.dll at 75E00000
--- Process 44976 thread 27480 exited with status 0x0
--- Process 44976 thread 34476 exited with status 0x0
--- Process 44976 thread 11724 exited with status 0x0
--- Process 44976 loaded C:\Windows\SysWOW64\kernel.appcore.dll at 744F0000
--- Process 44976 loaded C:\Windows\SysWOW64\msvcrt.dll at 76000000
--- Process 44976 thread 27816 created
--- Process 44976 thread 27816 exited with status 0x1
--- Process 44976 exited with status 0x1
Command exited with non-zero status 1
        Command being timed: "strace -f shellcheck system_health_validation.sh"
        User time (seconds): 0.00
        System time (seconds): 0.01
        Percent of CPU this job got: 0%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 1m 25.40s
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 462848
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 1933
        Minor (reclaiming a frame) page faults: 0
        Voluntary context switches: 0
        Involuntary context switches: 0
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 65536
        Exit status: 1

[aeiplatform]

Exited with 1? With those utilities you'll find the issue.

So you are on Windows os, first of all for developing on that *** disable Windows Defender in Real Time this take so much of your computational power. I would suggest to you use WSL 2 as terminal.

[piq9117]

Any chance you have the script available online? I can try running shellcheck on it for comparison

[maximunited]

It's corporate, so can't share the script. Also cannot disable windows security stuff :(

[SkySkimmer]

shellcheck consumes multi GB of memory on https://github.com/ocaml/ocaml/blob/0f22c5aec7c9787e832838bca3439b855ca06306/configure (just under 22k lines, 608KB) (tested that particular commit because that's what I had checked out, I expect any other commit will have similar results)

• on the first 2018 lines it takes 290MB
• on the first 5006 lines it takes 1370MB
• on the first 7034 lines it takes 3350MB
• on the full thing it takes over 9GB then I killed it

(not round numbers of lines because if there's a parsing error it fails quickly so I have to find the end of the if/loop/subshell/etc)

Tested with head -n $nlines configure | /bin/time --format="%M" /bin/shellcheck --format checkstyle --shell sh --external-sources -

[firedes]

Tested on https://github.com/juewuy/ShellCrash/blob/1.8.0/scripts/clash.sh: shellcheck --format json1 --shell=sh --external-sources --source-path=/dir/ShellCrash-1.8.0/scripts /dir/ShellCrash-1.8.0/scripts/clash.sh, the check process will never end and exhausts all the memory until make my computer dead.

Investigation:

1. Remove the --external-sources, everything is ok.
2. Keep the --external-sources, modify clash.sh to a very simple snippet with 3 source commands, memory will be exhausted:

   
   #!/bin/sh

clashsh() { read -p "请输入对应数字 > " num if [ "$num" = 1 ]; then source $CRASHDIR/getdate.sh && clashlink elif [ "$num" = 2 ]; then source $CRASHDIR/getdate.sh && update elif [ "$num" = 3 ]; then source $CRASHDIR/getdate.sh && userguide fi }

clashsh

3. With 2 `source` commands, the process takes about 10 seconds and 2G memory and then ends.

[koalaman]

With d80fdfa9e8e73 you can specify # shellcheck extended-analysis=false to skip DFA in particularly large files (also works in rc files, and via --extended-analysis=false.